From Newsgroup: comp.lang.tcl


Hi,

I use a library to format and print debugging message which use also Linux-terminal color-code.

the CORE problem is that "string length .." is used to format the message and add additional information etc
BUT the "string length .." count the NON-visible color-code as well as visible chars that result in a MISS-format string

example: (color is not visible in this thread !! - but all is "green")

-> outTXT<multi-line>
-> | |DEBUG[1]: aa -> LIST | <- count color codes
-> | |DEBUG[1]: -> | a1 : 111 | <- count color codes
-> | |DEBUG[1]: -> | a2 : 222 | <- count color codes
-> | |DEBUG[1]: -> | a3 : 333 |


a color-code is something like

## ------------------------------------------------------------------------
## DEBUG helper
## Black 0;30 Dark Gray 1;30
## Red 0;31 Light Red 1;31
## Green 0;32 Light Green 1;32
## Brown/Orange 0;33 Yellow 1;33
## Blue 0;34 Light Blue 1;34
## Purple 0;35 Light Purple 1;35
## Cyan 0;36 Light Cyan 1;36
## Light Gray 0;37 White 1;37
##
## 8bit (256) colors: https://stackoverflow.com/questions/4842424/list-of-ansi-color-escape-sequences
variable CL_COLOR
set CL_COLOR(red) "[1;31m"
set CL_COLOR(green) "[1;32m"
set CL_COLOR(yellow) "[1;33m"
set CL_COLOR(blue) "[1;34m"
set CL_COLOR(purple) "[38;5;206m"
set CL_COLOR(cyan) "[1;36m"
set CL_COLOR(lightcyan) "[38;5;51m"
set CL_COLOR(white) "[1;37m"
set CL_COLOR(grey) "[38;5;254m"
set CL_COLOR(orange) "[38;5;202m"
set CL_COLOR(no) ""
variable CL_RESET "[0;m"


code:

set tstmsg "$lib_debug::CL_COLOR(red)123$lib_debug::CL_RESET"

puts "$tstmsg"
puts [string length $tstmsg]


result:

123 (the color of "123" is red)
15


there is something MISSING in TCL and this is the "string length ..." only count the !! VISIBLE !! chars


mfg ao
--- Synchronet 3.20a-Linux NewsLink 1.114

From Newsgroup: comp.lang.tcl


→ add the problem to TCL-wiki also: https://wiki.tcl-lang.org/page/BUG+%2D+%27string+length%27+count+also+NON+visible+chars

--- Synchronet 3.20a-Linux NewsLink 1.114

From Newsgroup: comp.lang.tcl

aotto1968 <aotto1968@t-online.de> wrote:

there is something MISSING in TCL and this is the "string length ..."
only count the !! VISIBLE !! chars

No. The solution is to measure the string first, then add the terminal escapes in a lower level, just before output to the terminal.

There is no need to complicate string length with such a requirement,
as it is has not ever been defined as measuring only 'visible'
characters. It has always measured "all" characters in the string.
And it should continue to do so.

All those terminal escapes are simply "characters in the string" which "increase its length". They only become "invisible" when the string is
output to a device that interprets them in that way. Output them to a
disk file and they appear as bytes in the file.

Secondly, what is "visible" varies depending upon what terminal is used
for output, so making this work for all cases is a deep rabbit hole
with potentially no bottom.
--- Synchronet 3.20a-Linux NewsLink 1.114

From Newsgroup: comp.lang.tcl

On 3/5/24 09:19, aotto1968 wrote:

→ add the problem to TCL-wiki also: https://wiki.tcl-lang.org/page/BUG+%2D+%27string+length%27+count+also+NON+visible+chars
Noted in said WIKI page that what characters are visible is *VERY*

device dependent and that this is not something TCL is missing but
rather the application.

--- Synchronet 3.20a-Linux NewsLink 1.114

From Newsgroup: comp.lang.tcl


This is funny → it seems to be a "serious" problem to "format" the "terminal-output" with
invisible ctrl-character(s)

I understand that at least a TERM environment variable have to be available to understand
what is a ctrl-char and what is a printing-char.


On 05.03.24 16:21, Rich wrote:

aotto1968 <aotto1968@t-online.de> wrote:

there is something MISSING in TCL and this is the "string length ..."
only count the !! VISIBLE !! chars

No. The solution is to measure the string first, then add the terminal escapes in a lower level, just before output to the terminal.

There is no need to complicate string length with such a requirement,
as it is has not ever been defined as measuring only 'visible'
characters. It has always measured "all" characters in the string.
And it should continue to do so.

All those terminal escapes are simply "characters in the string" which "increase its length". They only become "invisible" when the string is output to a device that interprets them in that way. Output them to a
disk file and they appear as bytes in the file.

Secondly, what is "visible" varies depending upon what terminal is used
for output, so making this work for all cases is a deep rabbit hole
with potentially no bottom.

--- Synchronet 3.20a-Linux NewsLink 1.114

From Newsgroup: comp.lang.tcl

On 3/5/24 13:02, aotto1968 wrote:

This is funny → it seems to be a "serious" problem to "format" the "terminal-output" with
invisible ctrl-character(s)

I understand that at least a TERM environment variable have to be
available to understand
what is a ctrl-char and what is a printing-char.

You are making the invalid assumption that all strings only are output
to a terminal -- for most application few if any strings are output to a terminal. Many times string are just used to send to a socket to
communicate with clients/servers.

On 05.03.24 16:21, Rich wrote:

aotto1968 <aotto1968@t-online.de> wrote:

there is something MISSING in TCL and this is the "string length ..."
only count the !!  VISIBLE !!  chars

No.  The solution is to measure the string first, then add the terminal
escapes in a lower level, just before output to the terminal.

There is no need to complicate string length with such a requirement,
as it is has not ever been defined as measuring only 'visible'
characters.  It has always measured "all" characters in the string.
And it should continue to do so.

All those terminal escapes are simply "characters in the string" which
"increase its length".  They only become "invisible" when the string is
output to a device that interprets them in that way.  Output them to a
disk file and they appear as bytes in the file.

Secondly, what is "visible" varies depending upon what terminal is used
for output, so making this work for all cases is a deep rabbit hole
with potentially no bottom.

--- Synchronet 3.20a-Linux NewsLink 1.114

From Newsgroup: comp.lang.tcl

On 3/5/2024 2:02 PM, aotto1968 wrote:

This is funny → it seems to be a "serious" problem to "format" the "terminal-output" with
invisible ctrl-character(s)

I understand that at least a TERM environment variable have to be
available to understand
what is a ctrl-char and what is a printing-char.

[string length] doesn't know about your intentions around how to handle strings you pass to it. If you want to skip color codes, you can easily modify your input. string-map is your friend here:

% string length [string map [list $lib_debug::CL_COLOR(red) {}
$lib_debug::CL_COLOR(green) {}
...
$lib_debug::CL_RESET {}] $tstmsg]

This should give you the results you are after.


--- Synchronet 3.20a-Linux NewsLink 1.114

From Newsgroup: comp.lang.tcl

saitology9 <saitology9@gmail.com> wrote:

On 3/5/2024 2:02 PM, aotto1968 wrote:

This is funny → it seems to be a "serious" problem to "format" the
"terminal-output" with
invisible ctrl-character(s)

I understand that at least a TERM environment variable have to be
available to understand
what is a ctrl-char and what is a printing-char.

[string length] doesn't know about your intentions around how to handle strings you pass to it. If you want to skip color codes, you can easily modify your input. string-map is your friend here:

% string length [string map [list $lib_debug::CL_COLOR(red) {}
$lib_debug::CL_COLOR(green) {}
...
$lib_debug::CL_RESET {}] $tstmsg]

This should give you the results you are after.

Or, the OP could measure the string *before* they wrap it with terminal control characters (which *do* make the string longer) for output to a terminal.

--- Synchronet 3.20a-Linux NewsLink 1.114

From Newsgroup: comp.lang.tcl

aotto1968 <aotto1968@t-online.de> wrote:

On 05.03.24 16:21, Rich wrote:

aotto1968 <aotto1968@t-online.de> wrote:

there is something MISSING in TCL and this is the "string length
..." only count the !! VISIBLE !! chars

No. The solution is to measure the string first, then add the
terminal escapes in a lower level, just before output to the
terminal.

All those terminal escapes are simply "characters in the string" which
"increase its length". They only become "invisible" when the string is
output to a device that interprets them in that way. Output them to a
disk file and they appear as bytes in the file.

Secondly, what is "visible" varies depending upon what terminal is used
for output, so making this work for all cases is a deep rabbit hole
with potentially no bottom.

This is funny → it seems to be a "serious" problem to "format" the "terminal-output" with invisible ctrl-character(s)

I understand that at least a TERM environment variable have to be
available to understand what is a ctrl-char and what is a
printing-char.

There is no way for Tcl to know if you plan to send that string to the terminal defined by the TERM variable.

You might do that. Or you might send it to a file, or your stdout
might be redirected to a file, or you might send it to a socket and
onward to another machine, where it might be output to a very different 'terminal' from the one where it was created.

If you want string lengths of strings with terminal control sequences
ignored, then simply measure your strings *before* you wrap them with
terminal control sequences.

--- Synchronet 3.20a-Linux NewsLink 1.114

From Newsgroup: comp.lang.tcl


I understand the the default `string length ...` is the wrong place to add this feature
*but* there could be other options.

1. string -visible length "..." or "string -terminal length"
2. or add a "terminal command for all "terminal" related functions like color etc
→ terminal string length "..."
→ terminal color ...
3. your already have such a thing like `string is control ..` or `string is ...` character classes

and yes you can always do a massive code-reorganization to get a simple `string length …` working
without invisible character chars count.

a separate C-library for this task would be good because all other languages seems to have the same "problem"
using the "terminal" proper.

On 05.03.24 21:42, Rich wrote:

aotto1968 <aotto1968@t-online.de> wrote:

On 05.03.24 16:21, Rich wrote:

aotto1968 <aotto1968@t-online.de> wrote:

there is something MISSING in TCL and this is the "string length
..." only count the !! VISIBLE !! chars

No. The solution is to measure the string first, then add the
terminal escapes in a lower level, just before output to the
terminal.

All those terminal escapes are simply "characters in the string" which
"increase its length". They only become "invisible" when the string is
output to a device that interprets them in that way. Output them to a
disk file and they appear as bytes in the file.

Secondly, what is "visible" varies depending upon what terminal is used
for output, so making this work for all cases is a deep rabbit hole
with potentially no bottom.

This is funny → it seems to be a "serious" problem to "format" the
"terminal-output" with invisible ctrl-character(s)

I understand that at least a TERM environment variable have to be
available to understand what is a ctrl-char and what is a
printing-char.

There is no way for Tcl to know if you plan to send that string to the terminal defined by the TERM variable.

You might do that. Or you might send it to a file, or your stdout
might be redirected to a file, or you might send it to a socket and
onward to another machine, where it might be output to a very different 'terminal' from the one where it was created.

If you want string lengths of strings with terminal control sequences ignored, then simply measure your strings *before* you wrap them with terminal control sequences.

--- Synchronet 3.20a-Linux NewsLink 1.114

From Newsgroup: comp.lang.tcl

In article <us9epm$amac$1@dont-email.me>,
aotto1968 <aotto1968@t-online.de> wrote:

I understand the the default `string length ...` is the wrong place to
add this feature
*but* there could be other options.

1. string -visible length "..." or "string -terminal length"
2. or add a "terminal command for all "terminal" related functions like
color etc
→ terminal string length "..."
→ terminal color ...
3. your already have such a thing like `string is control ..` or `string
is ...` character classes

and yes you can always do a massive code-reorganization to get a simple >`string length …` working
without invisible character chars count.

a separate C-library for this task would be good because all other
languages seems to have the same "problem"
using the "terminal" proper.

There is such a C library. It's called "curses"

It appears there is a tcllib package "term" which does similar things,
though I have never tried it.
--
columbiaclosings.com
What's not in Columbia anymore..
--- Synchronet 3.20a-Linux NewsLink 1.114

From Newsgroup: comp.lang.tcl

On Tue, 5 Mar 2024 16:03:45 +0100
aotto1968 <aotto1968@t-online.de> wrote:

Hi,

I use a library to format and print debugging message which use also Linux-terminal color-code.

the CORE problem is that "string length .." is used to format the message and add additional information etc
BUT the "string length .." count the NON-visible color-code as well as visible chars that result in a MISS-format string

A simple helper proc is enough:

proc termlen {str} {
string length [regsub -all {\u001b\[[0-9;]+m} $str {}]
}
set CL_COLOR(red) "\u001b\[1;31m"
set CL_RESET "\u001b\[0;m"

set msg "This is a test!!"
set tstmsg $CL_COLOR(red)$msg$CL_RESET
puts $tstmsg

puts "msg length : [string length $msg]"
puts "tstmsg length : [string length $tstmsg]"
puts "termlen : [termlen $tstmsg]"

Output:

This is a test!! (output in red)
msg length : 16
tstmsg length : 28
termlen : 16

Also, be sure that the channel you are writing to is a terminal and no
a regular file; on *nix terminals at least this check works (channels
have a -mode option when the C function isatty(fd) returns true).

proc isterminal {chan} {expr {![catch {chan configure $chan -mode}]}}

~$ tclsh 2>/dev/null
% proc isterminal {chan} {expr {![catch {chan configure $chan -mode}]}}
% isterminal stdout
1
% isterminal stderr
0
--
Emiliano
--- Synchronet 3.20a-Linux NewsLink 1.114

From Newsgroup: comp.lang.tcl

On 3/6/24 03:59, aotto1968 wrote:

I understand the the default `string length ...` is the wrong place to
add this feature
*but* there could be other options.

1. string -visible length "..." or "string -terminal length"
2. or add a "terminal command for all "terminal" related functions like color etc
   → terminal string length "..."
   → terminal color ...
3. your already have such a thing like `string is control ..` or `string
is ...` character classes

and yes you can always do a massive code-reorganization to get a simple `string length …` working
without invisible character chars count.

a separate C-library for this task would be good because all other
languages seems to have the same "problem"
using the "terminal" proper.

Please feel free to code up and contribute, via a TIP, your suggestions.

On 05.03.24 21:42, Rich wrote:

aotto1968 <aotto1968@t-online.de> wrote:

On 05.03.24 16:21, Rich wrote:

aotto1968 <aotto1968@t-online.de> wrote:

there is something MISSING in TCL and this is the "string length
..." only count the !!  VISIBLE !!  chars

No.  The solution is to measure the string first, then add the
terminal escapes in a lower level, just before output to the
terminal.

All those terminal escapes are simply "characters in the string" which >>>> "increase its length".  They only become "invisible" when the string is >>>> output to a device that interprets them in that way.  Output them to a >>>> disk file and they appear as bytes in the file.

Secondly, what is "visible" varies depending upon what terminal is used >>>> for output, so making this work for all cases is a deep rabbit hole
with potentially no bottom.

This is funny → it seems to be a "serious" problem to "format" the
"terminal-output" with invisible ctrl-character(s)

I understand that at least a TERM environment variable have to be
available to understand what is a ctrl-char and what is a
printing-char.

There is no way for Tcl to know if you plan to send that string to the
terminal defined by the TERM variable.

You might do that.  Or you might send it to a file, or your stdout
might be redirected to a file, or you might send it to a socket and
onward to another machine, where it might be output to a very different
'terminal' from the one where it was created.

If you want string lengths of strings with terminal control sequences
ignored, then simply measure your strings *before* you wrap them with
terminal control sequences.

--- Synchronet 3.20a-Linux NewsLink 1.114

From Newsgroup: comp.lang.tcl

This thread reminded me of my movie subtitle program.

Subtitles have a limit on number of characters per line. My program
has to count characters all the time and enforce the limit. And it
turns out that some subtitle formats provide for phrases to be possibly rendered in italics with run-of-the-mill <i>HTML formatting</i>.

Of course, the HTML tags are not visible on the screen so they must
not be counted.

So my code reads the line into a separate variable, strips the HTML
code and counts the characters in that separate variable.

Seems to be pretty much the same case at hand here.
--
Luc

--- Synchronet 3.20a-Linux NewsLink 1.114

From Newsgroup: comp.lang.tcl

I update the wiki: https://wiki.tcl-lang.org/page/BUG+%2D+%27string+length%27+count+also+NON+visible+chars

update 7 mar 2024
The CORE problem is not just the string length the CORE problem is the representation of the string in tcl

(and all other programming languages as well)
Let's start simple: the problem is just the same as the multibyte-chars-problem raised up ~30 years ago.

at the beginning it was only ascii and a string was char* after a while they figure out that are languages with more chars than
plain ascii.
It took ~20 years to implement utf8 and finally utf16 into all programming languages.
tcl uses utf8 as script-encoding and utf16 internal as string representation because

a string is only useful if index operations can be performt like: string index $str 0 and utf16 is index-able and utf8 not.
The string length problem is not alone, there is also the index-operation-problem like:

string index or
string range or
format %40s
that doesn't work.

The CORE problem is that the terminal-chars are embedded into the string and not as an character attribute.

* the terminal is the main presentation-layer of a tcl. * the gui-toolkit is the main presentation-layer of tk.

The current string length in tcl is doing something in-between string bytelength and string visual-length because
multibyte-chars are reduced to a single-char but invisible console-ctrl-char are counted like a char.

solution
The solution is the same solution like in utf16 or tk. every char have to be presented by a data-structure that hold

the multibyte-char (utf16)
the terminal-encoding (color,bold,underline,...)
I call this utf32

The puts operate as:

connected to a terminal just translate the utf32 string into a stream understood by the terminal.
connected to a file just do a utf32 write into a file
(perhaps the utf32 can be encoded into utf8)
finally you can write a terminal-colored-string into a file and read it again with the color information still available.
I understand that this is not a job for tcl alone because all programming languages are involved and that is the reason I assume
a utf32 consortium is good and programmers of all languages start together to implement this shit.



On 05.03.24 16:03, aotto1968 wrote:

Hi,

I use a library to format and print debugging message which use also Linux-terminal color-code.

the CORE problem is that "string length .." is used to format the message and add additional information etc
BUT the "string length .." count the NON-visible color-code as well as visible chars that result in a MISS-format string

example: (color is not visible in this thread !! - but all is "green")

; outTXT<multi-line>
t;   |   |DEBUG[1]: aa   -> LIST                                     |  <- count color codes
t;   |   |DEBUG[1]:      ->   | a1                   : 111           |  <- count color codes
t;   |   |DEBUG[1]:      ->   | a2                   : 222           |  <- count color codes
t;   |   |DEBUG[1]:      ->   | a3                   : 333 |

a color-code is something like

  ## ------------------------------------------------------------------------
  ## DEBUG helper
  ##  Black        0;30     Dark Gray     1;30
  ##  Red          0;31     Light Red     1;31
  ##  Green        0;32     Light Green   1;32
  ##  Brown/Orange 0;33     Yellow        1;33
  ##  Blue         0;34     Light Blue    1;34
  ##  Purple       0;35     Light Purple  1;35
  ##  Cyan         0;36     Light Cyan    1;36
  ##  Light Gray   0;37     White         1;37
  ##
  ##  8bit (256) colors: https://stackoverflow.com/questions/4842424/list-of-ansi-color-escape-sequences
  variable CL_COLOR
  set  CL_COLOR(red)        "[1;31m"
  set  CL_COLOR(green)      "[1;32m"
  set  CL_COLOR(yellow)     "[1;33m"
  set  CL_COLOR(blue)       "[1;34m"
  set  CL_COLOR(purple)     "[38;5;206m"
  set  CL_COLOR(cyan)       "[1;36m"
  set  CL_COLOR(lightcyan)  "[38;5;51m"
  set  CL_COLOR(white)      "[1;37m"
  set  CL_COLOR(grey)       "[38;5;254m"
  set  CL_COLOR(orange)     "[38;5;202m"
  set  CL_COLOR(no)         ""
  variable  CL_RESET        "[0;m"

code:

set tstmsg  "$lib_debug::CL_COLOR(red)123$lib_debug::CL_RESET"

puts "$tstmsg"
puts [string length $tstmsg]

result:

123        (the color of "123" is red)
15

there is something MISSING in TCL and this is the "string length ..." only count the !! VISIBLE !! chars

mfg ao

--- Synchronet 3.20a-Linux NewsLink 1.114

From Newsgroup: comp.lang.tcl

In article <usc1bu$vli2$1@dont-email.me>,
aotto1968 <aotto1968@t-online.de> wrote:

I update the wiki: >https://wiki.tcl-lang.org/page/BUG+%2D+%27string+length%27+count+also+NON+visible+chars

update 7 mar 2024
The CORE problem is not just the string length the CORE problem is the >representation of the string in tcl

(and all other programming languages as well)
Let's start simple: the problem is just the same as the >multibyte-chars-problem raised up ~30 years ago.

at the beginning it was only ascii and a string was char* after a while
they figure out that are languages with more chars than
plain ascii.
It took ~20 years to implement utf8 and finally utf16 into all
programming languages.
tcl uses utf8 as script-encoding and utf16 internal as string
representation because

a string is only useful if index operations can be performt like: string >index $str 0 and utf16 is index-able and utf8 not.
The string length problem is not alone, there is also the >index-operation-problem like:

string index or
string range or
format %40s
that doesn't work.

The CORE problem is that the terminal-chars are embedded into the string
and not as an character attribute.

* the terminal is the main presentation-layer of a tcl. * the
gui-toolkit is the main presentation-layer of tk.

the terminal is the main presentation-layer of a tcl

But it's not. At this point, I would guess sockets are the main
"presentation layer", certainly the REST work I have been doing over
the past few years does no terminal I/O. If you want to talk to a
terminal use curses or whatever. That has no relation to the use
cases for files & sockets.
--
columbiaclosings.com
What's not in Columbia anymore..
--- Synchronet 3.20a-Linux NewsLink 1.114

From Newsgroup: comp.lang.tcl


If you interact with tcl, read/write debugging messages etc it is always a terminal or a gui application.
the "socket" is just like the "file" an other target. I understand that sending "additional-terminal-attributes"
over the socket is *not* a all-day behavior because:

1. you don't use *tcl* for that
2. there is already a *ssh* application
3. there are format(s) like *html* etc which add attributes to strings.

the "terminal-library" as you mentioned does a much but mostly not basic "string-index" operations
because this is the target of the programming language.

→ try to find a [string range $str 17 45] in a curses library etc.


On 07.03.24 14:51, Ted Nolan <tednolan> wrote:

In article <usc1bu$vli2$1@dont-email.me>,
aotto1968 <aotto1968@t-online.de> wrote:

I update the wiki:
https://wiki.tcl-lang.org/page/BUG+%2D+%27string+length%27+count+also+NON+visible+chars

update 7 mar 2024
The CORE problem is not just the string length the CORE problem is the
representation of the string in tcl

(and all other programming languages as well)
Let's start simple: the problem is just the same as the
multibyte-chars-problem raised up ~30 years ago.

at the beginning it was only ascii and a string was char* after a while
they figure out that are languages with more chars than
plain ascii.
It took ~20 years to implement utf8 and finally utf16 into all
programming languages.
tcl uses utf8 as script-encoding and utf16 internal as string
representation because

a string is only useful if index operations can be performt like: string
index $str 0 and utf16 is index-able and utf8 not.
The string length problem is not alone, there is also the
index-operation-problem like:

string index or
string range or
format %40s
that doesn't work.

The CORE problem is that the terminal-chars are embedded into the string
and not as an character attribute.

* the terminal is the main presentation-layer of a tcl. * the
gui-toolkit is the main presentation-layer of tk.

the terminal is the main presentation-layer of a tcl

But it's not. At this point, I would guess sockets are the main "presentation layer", certainly the REST work I have been doing over
the past few years does no terminal I/O. If you want to talk to a
terminal use curses or whatever. That has no relation to the use
cases for files & sockets.

--- Synchronet 3.20a-Linux NewsLink 1.114