
read_token(+Stream, -Token, -Class)

   Succeeds if the next token from the input stream Stream is successfully
read and unified with Token and its token class with Class.



Arguments
   Stream              Stream handle or alias (atom)
   Token               Variable or constant.
   Class               Variable or atom.

Type
   Character I/O

Description
   This predicate is an interface to the ECLiPSe tokenizer.  It can be used
   to read terms which are not ended by a fullstop or even to build whole
   new parsers.  The next token from the input stream Stream is read and
   unified with Token.  The token class of this token is unified with
   Class.


   The possible token classes with examples:



   ---------------------------------------
   | Input Example  Token    Class        |
   |------------------------------------  |
   | X              "X"      var          |
   | _              "_"      anonymous    |
   | abc            'abc'    atom         |
   | 'a-b'          'a-b'    quoted_atom  |
   | 123            123      integer      |
   | 1.2            1.2      float        |
   | 1_3            1_3      rational     |
   | 0.9__1.1       0.9__1.1 breal        |
   | "abc"          "abc"    string       |
   | |              "|"      solo         |
   | )              ")"      solo         |
   | (              "("      solo         |
   | <SPACE>(       "("      open_par     |
   | ,              ','      comma        |
   | .<NL>          '.'      fullstop     |
   | 1e789<NL>      "1e789"  error        |
   ---------------------------------------|

   Note that round, square and curly brackets are solo tokens whose
   value is returned as a string.  Opening brackets preceded by space
   are treated specially as open_par tokens.  Comma and fullstop have
   their own token class.  All syntax errors are reported as class
   error, with the input string up to the error as Token.  The default
   error handler for the event 190 (reading EOF) returns end_of_file
   in both Class and Token.

   Note about signed numbers: the tokenizer returns a sign followed by a
   number as two separate tokens.  For instance, the input "-5" is read
   as two tokens, the atom '-' and the integer 5.  In the case of bounded
   reals, this leads to "-1.1__-0.9" being returned as the atom '-' and
   the bounded real 1.1__-0.9.  This is a non-canonical breal, since the
   upper bound is lower than the lower bound.  read_token/2,3 is the only
   predicate that can produce such objects, and it is the programmer's
   responsibility to construct a valid breal using breal_bounds/3 and
   breal_from_bounds/3, taking the sign into account.  Note that all
   other arithmetic operations are undefined on non-canonical breals.



Modes and Determinism
   read_token(+, -, -) is det

Modules
   This predicate is sensitive to its module context (tool predicate, see @/2).

Exceptions
     4 --- Stream is not instantiated.
     5 --- Stream is not an atom or a stream handle.
     5 --- Class does not unify with an atom.
   190 --- End of file was encountered before reading any character.
   192 --- Stream is not an input stream.
   193 --- Stream is an illegal stream specification.
   198 --- Trying to read even after the error 190 was raised.

Examples
   
Success:
      ?- read_token(input,T,C).
              []
      T = []
      C = atom
      ?- read_token(input,T,C).
              [
      T = "["
      C = solo
      ?- read_token(input, "k",C).
              "k"
      C = string
      ?- read_token(input,T,C).
              X
      T = "X"
      C = var
      ?- read_token(input,T,C).
              1.6e-5.
      T = 1.6e-05
      C = float

Fail:
      ?- read_token(input, "[", C).
              &
      no.

Error:
      ?- read_token(input, T, C).
              ^D
      T = end_of_file
      C = end_of_file
      yes. (Error 190, default handler)

      read_token(S, a(b,c), C).         (Error 4).
      read_token("string", a(b,c), C).  (Error 5).
      read_token(9, X + 2, C).          (Error 192). % stream not open
      read_token(atom, X + 2, C).       (Error 193).






See Also
   get_chtab / 2, set_chtab / 2, read_token / 2
