From Newsgroup: comp.lang.tcl

I need to run two external processes (on Linux):

pdftotext -tsv one.pdf
pdftotext -tsv two.pdf

For each one I need to acquire the output and post-process it.
Both are completely independent.
(However, once I've finished post-processing I then do some work on
both sets of post-processed data together.)

Each external process takes about 3 secs so it takes just over 6 secs
to acquire the data from both processes.

When I've done something similar in Python I've used the multiprocessing
module and this has got my runtime close to the 3 secs.

In my experiments with Tcl's threading I've found the threading startup overhead to be rather large.

What is the fastest way to run two independent processes concurrently
and acquire their outputs using Tcl?
--- Synchronet 3.21f-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On Wed, 29 Apr 2026 07:38:23 -0000 (UTC), Mark Summerfield wrote:

I need to run two external processes (on Linux):

pdftotext -tsv one.pdf
pdftotext -tsv two.pdf

For each one I need to acquire the output and post-process it.
Both are completely independent.
(However, once I've finished post-processing I then do some work on
both sets of post-processed data together.)

Each external process takes about 3 secs so it takes just over 6 secs
to acquire the data from both processes.

When I've done something similar in Python I've used the multiprocessing module and this has got my runtime close to the 3 secs.

In my experiments with Tcl's threading I've found the threading startup overhead to be rather large.

What is the fastest way to run two independent processes concurrently
and acquire their outputs using Tcl?

Here's my serial version:

proc app::serial {pdftotext pdf1 pdf2} {
puts serial
set pdf1tsv [exec $pdftotext -tsv $pdf1 -]
set pdf2tsv [exec $pdftotext -tsv $pdf2 -]
list $pdf1tsv $pdf2tsv
}

This takes ~2 sec for two ~650 page PDFs.

With some help from Gemini (after I got past non-working and slow
solutions) I did a multiprocessing version:

proc app::multiprocess {pdftotext pdf1 pdf2} {
set p1 [open "|$pdftotext -tsv $pdf1 - 2>@1" r]
try {
set p2 [open "|$pdftotext -tsv $pdf2 - 2>@1" r]
try {
fconfigure $p1 -blocking 0
fconfigure $p2 -blocking 0
set pdf1tsv ""
set pdf2tsv ""
while {![eof $p1] || ![eof $p2]} {
append pdf1tsv [read $p1]
append pdf2tsv [read $p2]
after 1
}
} finally {
close $p2
}
} finally {
close $p1
}
list $pdf1tsv $pdf2tsv
}

This takes ~1 sec.
--- Synchronet 3.21f-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

Am 29.04.2026 um 10:51 schrieb Mark Summerfield:

This takes ~1 sec.

So it's 2x faster, as expected.
What's the issue?
--- Synchronet 3.21f-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On Wed, 29 Apr 2026 11:24:34 +0200, meshparts wrote:

Am 29.04.2026 um 10:51 schrieb Mark Summerfield:

This takes ~1 sec.

So it's 2x faster, as expected.
What's the issue?

When I originally asked I only had the serial approach.
I replied to myself once I had the multiprocessing approach which
solved the problem so that people could see it was solved.
--- Synchronet 3.21f-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

* Mark Summerfield <m.n.summerfield@gmail.com>
| With some help from Gemini (after I got past non-working and slow
| solutions) I did a multiprocessing version:

| proc app::multiprocess {pdftotext pdf1 pdf2} {
| set p1 [open "|$pdftotext -tsv $pdf1 - 2>@1" r]
| try {
| set p2 [open "|$pdftotext -tsv $pdf2 - 2>@1" r]
| try {
| fconfigure $p1 -blocking 0
| fconfigure $p2 -blocking 0

Depending on the output of $pdftotext, some -encoding option might be necessary, too.

| set pdf1tsv ""
| set pdf2tsv ""
| while {![eof $p1] || ![eof $p2]} {
| append pdf1tsv [read $p1]
| append pdf2tsv [read $p2]
| after 1
| }

I don't like the busy-waiting loop for eof, but a solution using
fileevents would require namespace vars or globals to collect the output
and signallig 'done', so ymmv.

R'
--- Synchronet 3.21f-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl


I don't understand why Threads are not used (in particular Thread Pools)

Here's my solution. Please tell me if there's a significant speed penalty.

# ===============================

package require Thread

# run up to 3 parallel workers; extra jobs are queued
set mytpool [tpool::create -minworkers 3]

# ..
set jobs {
"exec $pdftotext -tsv $pdf1 - 2>@1"
"exec $pdftotext -tsv $pdf2 - 2>@1"
"exec $pdftotext -tsv $pdf3 - 2>@1"
}

set T0 [clock milliseconds]
set myjobIDs {}
# scheduled all jobs
foreach job $jobs {
lappend myjobIDs [tpool::post -nowait $mytpool $job]
}
unset RESULT
puts "waiting for RESULT..."
while { [llength $myjobIDs] > 0 } {
# get the completed jobs; myjobIDs is updated with the list of the still pending jobs
set completedJobs [tpool::wait $mytpool $myjobIDs myjobIDs]
foreach job $completedJobs {
puts "== Job $job completed at [expr {[clock milliseconds]-$T0}] msec"
set RESULT($job) [tpool::get $mytpool $job]
}
}

puts "Result saved in the RESULT() array"
puts "Total processing time: [expr {[clock milliseconds]-$T0}] msec"
--- Synchronet 3.21f-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

* abu <user13892@newsgrouper.org.invalid>
| I don't understand why Threads are not used (in particular Thread Pools)

Most probably because Mark stated in Message-ID: <10sschf$3nvs2$1@dont-email.me>

In my experiments with Tcl's threading I've found the threading
startup overhead to be rather large.

| Here's my solution. Please tell me if there's a significant speed penalty.

Did you compare your version to Mark's solution? This would be the best comparison when running on the same hardware...

R'
--- Synchronet 3.21f-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

I created a tiny test program (65 LOC; shown at the end) to
compare timings. I did multiple timings and here're the averages:

serial (2 LOC) 2.020 sec
multiprocess (19 LOC) 1.055 sec
threaded (13 LOC) 1.061 sec

Since the difference between the multiprocess and threaded
approaches is so small and that the threaded code is simpler
and more appealing, I'm going to use the threaded version in
my programs (which only ever work with two PDFs at a time)
— so thank you "abu"!

#!/usr/bin/env tclsh9
# usage: time ./concurrent.tcl <s|m|t> <file1.pdf> <file2.pdf>

package require thread

proc main {} {
set pdftotext [auto_execok pdftotext]
set pdf1 [lindex $::argv 1]
set pdf2 [lindex $::argv 2]
switch [lindex $::argv 0] {
s { serial $pdftotext $pdf1 $pdf2 }
m { multiprocess $pdftotext $pdf1 $pdf2 }
t { threaded $pdftotext $pdf1 $pdf2 }
}
}

proc serial {pdftotext pdf1 pdf2} {
puts -nonewline "serial "
set tsv1 [exec $pdftotext -tsv $pdf1 - 2>@1]
set tsv2 [exec $pdftotext -tsv $pdf2 - 2>@1]
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

proc multiprocess {pdftotext pdf1 pdf2} {
puts -nonewline multiprocess
set p1 [open "|$pdftotext -tsv $pdf1 - 2>@1" r]
try {
set p2 [open "|$pdftotext -tsv $pdf2 - 2>@1" r]
try {
fconfigure $p1 -blocking 0
fconfigure $p2 -blocking 0
set tsv1 ""
set tsv2 ""
while {![eof $p1] || ![eof $p2]} {
append tsv1 [read $p1]
append tsv2 [read $p2]
after 1
}
} finally {
close $p2
}
} finally {
close $p1
}
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

proc threaded {pdftotext pdf1 pdf2} {
puts -nonewline "threaded "
set pool [tpool::create -minworkers 2]
set job1 [tpool::post -nowait $pool "exec $pdftotext -tsv $pdf1 - 2>@1"]
set job2 [tpool::post -nowait $pool "exec $pdftotext -tsv $pdf2 - 2>@1"]
set job_ids [list $job1 $job2]
while {[llength $job_ids] > 0} {
foreach job_id [tpool::wait $pool $job_ids job_ids] {
if {$job_id eq $job1} {
set tsv1 [tpool::get $pool $job_id]
} else {
set tsv2 [tpool::get $pool $job_id]
}
}
}
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

main
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl


Mark Summerfield <m.n.summerfield@gmail.com> posted:

I need to run two external processes (on Linux):

pdftotext -tsv one.pdf
pdftotext -tsv two.pdf

I am not an expert, but the construction (with Tcl 9.x) :

1) launch both processes in background

2) check the status with ::tcl::process

3) post-process the output of each process as soon as it has ended (*)

seems doable but no one mentions something similar, is this a construction
to avoid ?

(*) with a monolithic script if it is fast, I mean no thread or
different interpreters
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

* Mark Summerfield <m.n.summerfield@gmail.com>
| I created a tiny test program (65 LOC; shown at the end) to
| compare timings. I did multiple timings and here're the averages:

| serial (2 LOC) 2.020 sec
| multiprocess (19 LOC) 1.055 sec
| threaded (13 LOC) 1.061 sec

| Since the difference between the multiprocess and threaded
| approaches is so small and that the threaded code is simpler
| and more appealing, I'm going to use the threaded version in
| my programs (which only ever work with two PDFs at a time)
| — so thank you "abu"!

I wonder: you stated in your initial message

Message-ID: <10sschf$3nvs2$1@dont-email.me>
In my experiments with Tcl's threading I've found the threading
startup overhead to be rather large.

Can you tell what is/was the difference to the current solution which
obviously has no "startup overhead"?

R'
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On Wed, 29 Apr 2026 07:38:23 -0000 (UTC)
Mark Summerfield <m.n.summerfield@gmail.com> wrote:

I need to run two external processes (on Linux):

pdftotext -tsv one.pdf
pdftotext -tsv two.pdf

You can use pipes and run the processes in the background, collecting output with the event loop. Here's a rough draft

proc runit {var file} {
lassign [chan pipe] cr cw
exec pdftotext -tsv $file - >@ $cw &
chan close $cw
chan configure $cr -blocking 0
chan event $cr readable [list handle $var $cr]
}
proc handle {var fd} {
global $var
append $var [chan read $fd]
if {[chan eof $fd]} {
chan close $fd
set ::done 1
}
}
puts "sequential: [time {
set out1 [exec pdftotext -tsv one.pdf -]
set out2 [exec pdftotext -tsv two.pdf -]
puts "one.pdf [string length $out1]"
puts "two.pdf [string length $out2]"
}]"
puts "parallel: [time {
runit out1 one.pdf
runit out2 two.pdf
vwait done
vwait done
puts "one.pdf [string length $out1]"
puts "two.pdf [string length $out2]"
}]"


Regards
--
Emiliano
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On Fri, 01 May 2026 22:54:54 +0200, Ralf Fassel wrote:

* Mark Summerfield <m.n.summerfield@gmail.com>
| I created a tiny test program (65 LOC; shown at the end) to
| compare timings. I did multiple timings and here're the averages:

| serial (2 LOC) 2.020 sec
| multiprocess (19 LOC) 1.055 sec
| threaded (13 LOC) 1.061 sec

| Since the difference between the multiprocess and threaded
| approaches is so small and that the threaded code is simpler
| and more appealing, I'm going to use the threaded version in
| my programs (which only ever work with two PDFs at a time)
| — so thank you "abu"!

I wonder: you stated in your initial message

Message-ID: <10sschf$3nvs2$1@dont-email.me>
In my experiments with Tcl's threading I've found the threading
startup overhead to be rather large.

Can you tell what is/was the difference to the current solution which obviously has no "startup overhead"?

R'

Yes, the difference was that I started out using thread::create etc.,
rather than using tpool. I've put a new version that compares them
all at the end. Anyone can compare timings for themselves if they
have one or two big PDF files (the program needs two but for tests
it is fine if it is the same one).

On an old laptop:

serial (2 LOC) 6.37 sec
multiprocess (19 LOC) 3.33 sec
thread pool (15 LOC) 3.60 sec
threaded (22 LOC) 3.66 sec

I've now gone back to using the multiprocess version.
Here's the full test code.

#!/usr/bin/env tclsh9
# usage: time ./concurrent.tcl <s|m|p|t> <file1.pdf> <file2.pdf>

package require thread 3

const OPT -tsv ;# OR if not supported by older pdftotext use: -bbox
const PDFTOTEXT [auto_execok pdftotext]

proc main {} {
set pdf1 [lindex $::argv 1]
set pdf2 [lindex $::argv 2]
switch [lindex $::argv 0] {
s { serial $pdf1 $pdf2 }
m { multiprocess $pdf1 $pdf2 }
p { thread_pool $pdf1 $pdf2 }
t { threaded $pdf1 $pdf2 }
}
}

proc serial {pdf1 pdf2} {
puts -nonewline "serial "
set tsv1 [exec $::PDFTOTEXT $::OPT $pdf1 - 2>@1]
set tsv2 [exec $::PDFTOTEXT $::OPT $pdf2 - 2>@1]
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

proc multiprocess {pdf1 pdf2} {
puts -nonewline multiprocess
set p1 [open "|$::PDFTOTEXT $::OPT $pdf1 - 2>@1" r]
try {
set p2 [open "|$::PDFTOTEXT $::OPT $pdf2 - 2>@1" r]
try {
fconfigure $p1 -blocking 0
fconfigure $p2 -blocking 0
set tsv1 ""
set tsv2 ""
while {![eof $p1] || ![eof $p2]} {
append tsv1 [read $p1]
append tsv2 [read $p2]
after 1
}
} finally {
close $p2
}
} finally {
close $p1
}
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

proc thread_pool {pdf1 pdf2} {
puts -nonewline "thread pool "
set pool [tpool::create -minworkers 2]
set job1 [tpool::post -nowait $pool \
"exec $::PDFTOTEXT $::OPT $pdf1 - 2>@1"]
set job2 [tpool::post -nowait $pool \
"exec $::PDFTOTEXT $::OPT $pdf2 - 2>@1"]
set job_ids [list $job1 $job2]
while {[llength $job_ids] > 0} {
foreach job_id [tpool::wait $pool $job_ids job_ids] {
if {$job_id eq $job1} {
set tsv1 [tpool::get $pool $job_id]
} else {
set tsv2 [tpool::get $pool $job_id]
}
}
}
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

proc threaded {pdf1 pdf2} {
puts -nonewline "threaded "
set tid1 [thread::create -joinable]
set tid2 [thread::create -joinable]
tsv::set shared pdf1 $pdf1
tsv::set shared pdf2 $pdf2
tsv::set shared pdftotext $::PDFTOTEXT
tsv::set shared opt $::OPT
thread::send -async $tid1 {
tsv::set shared tsv1 \
[exec -encoding utf-8 {*}[tsv::get shared pdftotext] \
[tsv::get shared opt] [tsv::get shared pdf1] - 2>@1]
}
thread::send -async $tid2 {
tsv::set shared tsv2 \
[exec -encoding utf-8 {*}[tsv::get shared pdftotext] \
[tsv::get shared opt] [tsv::get shared pdf2] - 2>@1]
}
thread::release $tid1
thread::join $tid1
thread::release $tid2
thread::join $tid2
set tsv1 [tsv::get shared tsv1]
set tsv2 [tsv::get shared tsv2]
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

main
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

Shameless plug...

Bit late to the topic, but the simplest way to parallelize multiple
processes or threads and wait for completion is promises, if you do not
mind an external package. Bit of a learning curve however.

lappend promises [promise::pexec pdftotext pdf1.pdf pdf1.txt]
lappend promises [promise::pexec pdftotext pdf2.pdf pdf2.txt]
set waiter [promise::all $promises]
# Assumes eventloop not running!
promise::eventloop $waiter

Timing:

% time {demo} <- using promises
2606403 microseconds per iteration
% time {demo2} <- sequential exec's
4762417 microseconds per iteration

https://wiki.tcl-lang.org/page/promise
https://tcl-promise.magicsplat.com/ https://www.magicsplat.com/blog/tags/promises/

--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

* Ashok <apnmbx-public@yahoo.com>
| Shameless plug...

| Bit late to the topic, but the simplest way to parallelize multiple
| processes or threads and wait for completion is promises, if you do
| not mind an external package. Bit of a learning curve however. --<snip-snip>--
| https://tcl-promise.magicsplat.com/

Ashok,
since coroutines are already part of TCL, any chance of getting promises
into the core? It would seem to me as a 'natural' addition for async
features in TCL, and the package looks quite mature...

R'
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On Sat, 9 May 2026 16:03:16 +0530, Ashok wrote:

Shameless plug...

Bit late to the topic, but the simplest way to parallelize multiple processes or threads and wait for completion is promises, if you do not
mind an external package. Bit of a learning curve however.

lappend promises [promise::pexec pdftotext pdf1.pdf pdf1.txt]
lappend promises [promise::pexec pdftotext pdf2.pdf pdf2.txt]
set waiter [promise::all $promises]
# Assumes eventloop not running!
promise::eventloop $waiter

Timing:

% time {demo} <- using promises
2606403 microseconds per iteration
% time {demo2} <- sequential exec's
4762417 microseconds per iteration

https://wiki.tcl-lang.org/page/promise
https://tcl-promise.magicsplat.com/ https://www.magicsplat.com/blog/tags/promises/

I tried it but hit a problem. Here's the code I used:

proc promised {pdf1 pdf2} {
set p1 [promise::pexec $::PDFTOTEXT $::OPT $pdf1 - 2>@1]
set p2 [promise::pexec $::PDFTOTEXT $::OPT $pdf2 - 2>@1]
set waiter [promise::all [list $p1 $p2]]
# Assumes eventloop not running!
promise::eventloop $waiter
set tsv1 [$p1 getdata]
set tsv2 [$p2 getdata]
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

I used promise-1.2.0.tm. Here's the error:

$ ./concurrent.tcl P ~/commercial/pdfs/boson[12].pdf
invalid command name "::oo::Obj22"
while executing
"$p1 getdata"
(procedure "promised" line 7)
invoked from within
"promised $pdf1 $pdf2 "
(procedure "main" line 9)
invoked from within
"main"
(file "./concurrent.tcl" line 112)

--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

* Mark Summerfield <m.n.summerfield@gmail.com>
| > https://wiki.tcl-lang.org/page/promise
| > https://tcl-promise.magicsplat.com/
| > https://www.magicsplat.com/blog/tags/promises/

| I tried it but hit a problem. Here's the code I used:

| proc promised {pdf1 pdf2} {
| set p1 [promise::pexec $::PDFTOTEXT $::OPT $pdf1 - 2>@1]
| set p2 [promise::pexec $::PDFTOTEXT $::OPT $pdf2 - 2>@1]
| set waiter [promise::all [list $p1 $p2]]
| # Assumes eventloop not running!
| promise::eventloop $waiter
| set tsv1 [$p1 getdata]
| set tsv2 [$p2 getdata]

promise::eventloop already returns the result of the 'waiter' promise
(i.e. those registered in promise::all).

So change those two 'getdata' calls to

lassign [promise::eventloop $waiter] tsv1 tsv2

| puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"

HTH
R'
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On Tue, 12 May 2026 16:58:55 +0200, Ralf Fassel wrote:

* Mark Summerfield <m.n.summerfield@gmail.com>
| > https://wiki.tcl-lang.org/page/promise
| > https://tcl-promise.magicsplat.com/
| > https://www.magicsplat.com/blog/tags/promises/

| I tried it but hit a problem. Here's the code I used:

| proc promised {pdf1 pdf2} {
| set p1 [promise::pexec $::PDFTOTEXT $::OPT $pdf1 - 2>@1]
| set p2 [promise::pexec $::PDFTOTEXT $::OPT $pdf2 - 2>@1]
| set waiter [promise::all [list $p1 $p2]]
| # Assumes eventloop not running!
| promise::eventloop $waiter
| set tsv1 [$p1 getdata]
| set tsv2 [$p2 getdata]

promise::eventloop already returns the result of the 'waiter' promise
(i.e. those registered in promise::all).

So change those two 'getdata' calls to

lassign [promise::eventloop $waiter] tsv1 tsv2

| puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"

HTH
R'

Thanks, I've now done that. Here are the new timings (each is the best
of several):

sec method
2.010 serial
1.052 multiprocess
1.065 thread_pool
1.067 threaded
8.366 promised
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

Am 13.05.2026 um 10:54 schrieb Mark Summerfield:

Thanks, I've now done that. Here are the new timings (each is the best
of several):

sec method
2.010 serial
1.052 multiprocess
1.065 thread_pool
1.067 threaded
8.366 promised

So with promisses it's 4x slower than serial?
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

I would not be in support of this myself. As it is I'm skeptical of
adding packages to the core because there simply are not enough folks to maintain the packages already there.

Additionally, promises are still a "fringe" idiom in Tcl land and not
widely used or adopted.

/Ashok

On 5/11/2026 2:38 PM, Ralf Fassel wrote:

Ashok,
since coroutines are already part of TCL, any chance of getting promises
into the core? It would seem to me as a 'natural' addition for async features in TCL, and the package looks quite mature...

R'

--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

I'm surprised by the promises result below (not that I doubt it). I'll
have to take a look when I have some time.

In my tests that I posted earlier, the promise version took about the
same time as the multiprocess one.

The difference between my example and yours is that in my example,
pdftotext was writing to a file and not to its stdout. In your example,
it is writing back to the pipe and read directly in Tcl.

I wonder if the difference stems from your code essentially doing a busy
loop reading data while the promise version goes through the event loop
though I cannot explain why that would make that much difference.

Worth investigating further when I have time...

/Ashok

On 5/13/2026 2:24 PM, Mark Summerfield wrote:

Thanks, I've now done that. Here are the new timings (each is the best
of several):

sec method
2.010 serial
1.052 multiprocess
1.065 thread_pool
1.067 threaded
8.366 promised

--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On Thu, 14 May 2026 17:20:51 +0530, Ashok wrote:

I'm surprised by the promises result below (not that I doubt it). I'll
have to take a look when I have some time.

In my tests that I posted earlier, the promise version took about the
same time as the multiprocess one.

The difference between my example and yours is that in my example,
pdftotext was writing to a file and not to its stdout. In your example,
it is writing back to the pipe and read directly in Tcl.

I wonder if the difference stems from your code essentially doing a busy loop reading data while the promise version goes through the event loop though I cannot explain why that would make that much difference.

Worth investigating further when I have time...

/Ashok

On 5/13/2026 2:24 PM, Mark Summerfield wrote:

Thanks, I've now done that. Here are the new timings (each is the best
of several):

sec method
2.010 serial
1.052 multiprocess
1.065 thread_pool
1.067 threaded
8.366 promised

In the hope it helps, below is the full source for the example I used.
I ran it on Tcl/Tk 9.0.3 (64-bit), Debian GNU/Linux 12 (bookworm)
Linux 6.1.0-44-amd64 (x86_64), 12th Gen Intel Core i7-12700 20 cores.
I used two PDF files both of 647 pages and did several runs of each
method to find the best time.

#!/usr/bin/env tclsh9
# usage: time ./concurrent.tcl <s|m|p|t> <file1.pdf> <file2.pdf>

package require thread 3
tcl::tm::path add .
package require promise

const OPT -tsv ;# If older pdftotext doesn't support -tsv use -bbox
const PDFTOTEXT [auto_execok pdftotext]

proc main {} {
set pdf1 [lindex $::argv 1]
set pdf2 [lindex $::argv 2]
switch [lindex $::argv 0] {
h - -h - --help {
puts "usage: <s|m|p|t|P> <file1.pdf> <file2.pdf"
exit
}
s { serial $pdf1 $pdf2 }
m { multiprocess $pdf1 $pdf2 }
p { thread_pool $pdf1 $pdf2 }
t { threaded $pdf1 $pdf2 }
P { promised $pdf1 $pdf2 }
}
}

proc serial {pdf1 pdf2} {
puts -nonewline "serial "
set tsv1 [exec $::PDFTOTEXT $::OPT $pdf1 - 2>@1]
set tsv2 [exec $::PDFTOTEXT $::OPT $pdf2 - 2>@1]
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

proc multiprocess {pdf1 pdf2} {
puts -nonewline multiprocess
set p1 [open "|$::PDFTOTEXT $::OPT $pdf1 - 2>@1" r]
try {
set p2 [open "|$::PDFTOTEXT $::OPT $pdf2 - 2>@1" r]
try {
fconfigure $p1 -blocking 0
fconfigure $p2 -blocking 0
set tsv1 ""
set tsv2 ""
while {![eof $p1] || ![eof $p2]} {
append tsv1 [read $p1]
append tsv2 [read $p2]
after 1
}
} finally {
close $p2
}
} finally {
close $p1
}
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

proc thread_pool {pdf1 pdf2} {
puts -nonewline "thread pool "
set pool [tpool::create -minworkers 2]
set job1 [tpool::post -nowait $pool \
"exec $::PDFTOTEXT $::OPT $pdf1 - 2>@1"]
set job2 [tpool::post -nowait $pool \
"exec $::PDFTOTEXT $::OPT $pdf2 - 2>@1"]
set job_ids [list $job1 $job2]
while {[llength $job_ids] > 0} {
foreach job_id [tpool::wait $pool $job_ids job_ids] {
if {$job_id eq $job1} {
set tsv1 [tpool::get $pool $job_id]
} else {
set tsv2 [tpool::get $pool $job_id]
}
}
}
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

proc threaded {pdf1 pdf2} {
puts -nonewline "threaded "
set tid1 [thread::create -joinable]
set tid2 [thread::create -joinable]
tsv::set shared pdf1 $pdf1
tsv::set shared pdf2 $pdf2
tsv::set shared pdftotext $::PDFTOTEXT
tsv::set shared opt $::OPT
thread::send -async $tid1 {
tsv::set shared tsv1 \
[exec -encoding utf-8 {*}[tsv::get shared pdftotext] \
[tsv::get shared opt] [tsv::get shared pdf1] - 2>@1]
}
thread::send -async $tid2 {
tsv::set shared tsv2 \
[exec -encoding utf-8 {*}[tsv::get shared pdftotext] \
[tsv::get shared opt] [tsv::get shared pdf2] - 2>@1]
}
thread::release $tid1
thread::join $tid1
thread::release $tid2
thread::join $tid2
set tsv1 [tsv::get shared tsv1]
set tsv2 [tsv::get shared tsv2]
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

proc promised {pdf1 pdf2} {
set p1 [promise::pexec $::PDFTOTEXT $::OPT $pdf1 - 2>@1]
set p2 [promise::pexec $::PDFTOTEXT $::OPT $pdf2 - 2>@1]
set waiter [promise::all [list $p1 $p2]]
# Assumes eventloop not running!
lassign [promise::eventloop $waiter] tsv1 tsv2
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

main
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On 5/15/2026 4:08 PM, Mark Summerfield wrote:

In the hope it helps, below is the full source for the example I used.
I ran it on Tcl/Tk 9.0.3 (64-bit), Debian GNU/Linux 12 (bookworm)
Linux 6.1.0-44-amd64 (x86_64), 12th Gen Intel Core i7-12700 20 cores.
I used two PDF files both of 647 pages and did several runs of each
method to find the best time.

Thanks, having a benchmark source helped. However I cannot reproduce
your results. The promise version is as fast as any other. My laptop is
long in the tooth but that should not make a difference in comparative
terms I think.

Below is what I get using the following shell script:

------
#!/bin/sh

for method in s m p t P; do
for i in $(seq 1 5); do
time -p ~/tcl/9.0.3/x64/bin/tclsh9.0 bench.tcl $method x.pdf
y.pdf 2>&1 | tr '\n' ' '
# Appends to previous line!
echo "Method $method, Run $i"
done
echo "---------------------------------------------------------"
done
-----

I used -bbox instead of -tsv as my pdftotext does not support the
latter. Tests on my Ubuntu 22 WSL. All versions, including promises are
about twice as fast as the serial one. On every run, there seems to be
one or two exceptionally fast anomaly, independent of the method. Not
sure why that is, some fortunate cache or memory effect?

Here are the results, more or less as expected.

serial tsv1=3559848 tsv2=3559848 real 5.59 user 0.82 sys 0.27
Method s, Run 1
serial tsv1=3559848 tsv2=3559848 real 5.50 user 0.82 sys 0.29
Method s, Run 2
serial tsv1=3559848 tsv2=3559848 real 5.61 user 0.82 sys 0.26
Method s, Run 3
serial tsv1=3559848 tsv2=3559848 real 4.24 user 0.83 sys 0.25
Method s, Run 4
serial tsv1=3559848 tsv2=3559848 real 5.59 user 0.81 sys 0.27
Method s, Run 5
---------------------------------------------------------
multiprocess tsv1=3559849 tsv2=3559849 real 3.13 user 0.95 sys 0.33
Method m, Run 1
multiprocess tsv1=3559849 tsv2=3559849 real 3.05 user 0.89 sys 0.36
Method m, Run 2
multiprocess tsv1=3559849 tsv2=3559849 real 3.12 user 0.95 sys 0.31
Method m, Run 3
multiprocess tsv1=3559849 tsv2=3559849 real 3.13 user 0.96 sys 0.30
Method m, Run 4
multiprocess tsv1=3559849 tsv2=3559849 real 3.13 user 0.97 sys 0.30
Method m, Run 5
---------------------------------------------------------
thread pool tsv1=3559848 tsv2=3559848 real 3.21 user 0.93 sys 0.40
Method p, Run 1
thread pool tsv1=3559848 tsv2=3559848 real 3.15 user 0.90 sys 0.39
Method p, Run 2
thread pool tsv1=3559848 tsv2=3559848 real 1.79 user 0.94 sys 0.37
Method p, Run 3
thread pool tsv1=3559848 tsv2=3559848 real 3.14 user 0.97 sys 0.31
Method p, Run 4
thread pool tsv1=3559848 tsv2=3559848 real 3.10 user 0.90 sys 0.37
Method p, Run 5
---------------------------------------------------------
threaded tsv1=3559848 tsv2=3559848 real 3.17 user 0.90 sys 0.41
Method t, Run 1
threaded tsv1=3559848 tsv2=3559848 real 3.14 user 0.90 sys 0.39
Method t, Run 2
threaded tsv1=3559848 tsv2=3559848 real 3.14 user 0.90 sys 0.37
Method t, Run 3
threaded tsv1=3559848 tsv2=3559848 real 3.14 user 0.94 sys 0.31
Method t, Run 4
threaded tsv1=3559848 tsv2=3559848 real 3.14 user 0.92 sys 0.35
Method t, Run 5
---------------------------------------------------------
promise tsv1=3559849 tsv2=3559849 real 3.33 user 2.68 sys 0.42
Method P, Run 1
promise tsv1=3559849 tsv2=3559849 real 3.30 user 2.48 sys 0.46
Method P, Run 2
promise tsv1=3559849 tsv2=3559849 real 1.94 user 2.48 sys 0.46
Method P, Run 3
promise tsv1=3559849 tsv2=3559849 real 3.35 user 2.69 sys 0.39
Method P, Run 4
promise tsv1=3559849 tsv2=3559849 real 3.31 user 2.64 sys 0.41
Method P, Run 5
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

The most obvious difference is that I am running Linux on the hardware
and (I think) you are running Linux on Windows. All I can suggest is
trying the same test on Linux that's running directly on the hardware?
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On 5/18/2026 12:41 PM, Mark Summerfield wrote:

The most obvious difference is that I am running Linux on the hardware
and (I think) you are running Linux on Windows. All I can suggest is
trying the same test on Linux that's running directly on the hardware?

Don't have such a beast :-)

However, I would have thought all benchmarks would be similarly affected
if time measurements are questionable. I do find pretty consistent
results between running Tcl benchmarks directly on Windows versus within
WSL.

Perhaps I/O (WSL is slow) is the difference. Strange...

If you are not overly bored by this, could you post the result of the
script I included? I'm curious to see the variance on your system.

/Ashok
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

Hi Ashok,

I had to modify your script to this:

#!/bin/bash

for method in s m p t P; do
for i in $(seq 1 5); do
time -p ~/opt/tcl9/bin/tclsh9.0 \
~/app/tcl/xmisc/concurrent.tcl $method \
~/commercial/pdfs/summerfield-*pdf 2>&1 | tr '\n' ' '
# Appends to previous line!
echo "Method $method, Run $i"
done
echo "---------------------------------------------------------"
done

I had great trouble getting it working until I realised that I needed
to use bash not sh.

Here're the results:

$ ./concurrent-run.sh
serial tsv1=14809914 tsv2=14816735 real 2.07
user 2.05
sys 0.09
Method s, Run 1
serial tsv1=14809914 tsv2=14816735 real 2.06
user 2.03
sys 0.09
Method s, Run 2
serial tsv1=14809914 tsv2=14816735 real 2.06
user 1.99
sys 0.13
Method s, Run 3
serial tsv1=14809914 tsv2=14816735 real 2.05
user 2.03
sys 0.08
Method s, Run 4
serial tsv1=14809914 tsv2=14816735 real 2.06
user 2.02
sys 0.09
Method s, Run 5
---------------------------------------------------------
multiprocess tsv1=14809915 tsv2=14816736 real 1.05
user 2.03
sys 0.09
Method m, Run 1
multiprocess tsv1=14809915 tsv2=14816736 real 1.05
user 2.05
sys 0.07
Method m, Run 2
multiprocess tsv1=14809915 tsv2=14816736 real 1.05
user 2.03
sys 0.08
Method m, Run 3
multiprocess tsv1=14809915 tsv2=14816736 real 1.05
user 2.04
sys 0.08
Method m, Run 4
multiprocess tsv1=14809915 tsv2=14816736 real 1.05
user 2.02
sys 0.09
Method m, Run 5
---------------------------------------------------------
thread pool tsv1=14809914 tsv2=14816735 real 1.08
user 2.06
sys 0.11
Method p, Run 1
thread pool tsv1=14809914 tsv2=14816735 real 1.08
user 2.06
sys 0.11
Method p, Run 2
thread pool tsv1=14809914 tsv2=14816735 real 1.07
user 2.07
sys 0.09
Method p, Run 3
thread pool tsv1=14809914 tsv2=14816735 real 1.07
user 2.08
sys 0.09
Method p, Run 4
thread pool tsv1=14809914 tsv2=14816735 real 1.08
user 2.04
sys 0.13
Method p, Run 5
---------------------------------------------------------
threaded tsv1=14809914 tsv2=14816735 real 1.08
user 2.09
sys 0.09
Method t, Run 1
threaded tsv1=14809914 tsv2=14816735 real 1.08
user 2.08
sys 0.09
Method t, Run 2
threaded tsv1=14809914 tsv2=14816735 real 1.08
user 2.05
sys 0.12
Method t, Run 3
threaded tsv1=14809914 tsv2=14816735 real 1.08
user 2.05
sys 0.12
Method t, Run 4
threaded tsv1=14809914 tsv2=14816735 real 1.08
user 2.07
sys 0.10
Method t, Run 5
---------------------------------------------------------
promised tsv1=14809915 tsv2=14816736 real 8.53
user 10.39
sys 0.26
Method P, Run 1
promised tsv1=14809915 tsv2=14816736 real 8.47
user 10.46
sys 0.15
Method P, Run 2
promised tsv1=14809915 tsv2=14816736 real 8.52
user 10.41
sys 0.23
Method P, Run 3
promised tsv1=14809915 tsv2=14816736 real 8.50
user 10.43
sys 0.20
Method P, Run 4
promised tsv1=14809915 tsv2=14816736 real 8.51
user 10.42
sys 0.22
Method P, Run 5
---------------------------------------------------------
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

* Ashok <apnmbx-public@yahoo.com>
--<snip-snip>--
| time -p ~/tcl/9.0.3/x64/bin/tclsh9.0 bench.tcl
| threaded tsv1=3559848 tsv2=3559848 real 3.17 user 0.90 sys 0.41
| promise tsv1=3559849 tsv2=3559849 real 3.33 user 2.68 sys 0.42

* Mark Summerfield <m.n.summerfield@gmail.com>
--<snip-snip>--
| time -p ~/opt/tcl9/bin/tclsh9.0 \
~/app/tcl/xmisc/concurrent.tcl
| threaded tsv1=14809914 tsv2=14816735 real 1.08 user 2.09 sys 0.09
| promised tsv1=14809915 tsv2=14816736 real 8.53 user 10.39 sys 0.26

I bet there is some significant difference between bench.tcl and
concurrent.tcl which explains the huge difference between the 'threaded'
and 'promised' versions of Ashok and Mark.

Could both of you post the current version of your scripts,
so one could retry?

R'
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On 5/21/2026 8:21 AM, Ralf Fassel wrote:

I bet there is some significant difference between bench.tcl and concurrent.tcl which explains the huge difference between the 'threaded'
and 'promised' versions of Ashok and Mark.

Yes, the difference is significant. I suspect the difference is either
in pdftotext itself or how it is spawned:

- It is called with different arguments: "pdftotext pdf1.pdf" vs.
pdftotext pdf1.pdf pdf1.txt" so perhaps pdftotext behaves differently internally to handle output to stdout or to a file.

- It is invoked with exec vs. promised's pexec. Is pexec just a wrapper
or does it spend time setting things up which may slow it down?


How about replacing it with a simple fake app that does nothing and
timing it then?


--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On Thu, 21 May 2026 14:21:18 +0200, Ralf Fassel wrote:

* Ashok <apnmbx-public@yahoo.com>
--<snip-snip>--
| time -p ~/tcl/9.0.3/x64/bin/tclsh9.0 bench.tcl
| threaded tsv1=3559848 tsv2=3559848 real 3.17 user 0.90 sys 0.41
| promise tsv1=3559849 tsv2=3559849 real 3.33 user 2.68 sys 0.42

* Mark Summerfield <m.n.summerfield@gmail.com>
--<snip-snip>--
| time -p ~/opt/tcl9/bin/tclsh9.0 \
~/app/tcl/xmisc/concurrent.tcl
| threaded tsv1=14809914 tsv2=14816735 real 1.08 user 2.09 sys 0.09
| promised tsv1=14809915 tsv2=14816736 real 8.53 user 10.39 sys 0.26

I bet there is some significant difference between bench.tcl and concurrent.tcl which explains the huge difference between the 'threaded'
and 'promised' versions of Ashok and Mark.

Could both of you post the current version of your scripts,
so one could retry?

R'

Sure, below is concurrent.tcl.

I ran my tests using two very similar PDFs both 647 pages long,
one 6.6MB the other 6.5MB. For copyright reasons I can't share them.

#!/usr/bin/env tclsh9
# usage: time ./concurrent.tcl <s|m|p|t> <file1.pdf> <file2.pdf>

if {![catch {file readlink [info script]} name]} {
const APPPATH [file dirname $name]
} else {
const APPPATH [file normalize [file dirname [info script]]]
}
tcl::tm::path add $APPPATH

package require promise
package require thread 3

const OPT -tsv ;# If older pdftotext doesn't support -tsv use -bbox
const PDFTOTEXT [auto_execok pdftotext]

proc main {} {
set pdf1 [lindex $::argv 1]
set pdf2 [lindex $::argv 2]
switch [lindex $::argv 0] {
h - -h - --help {
puts "usage: <s|m|p|t|P> <file1.pdf> <file2.pdf"
exit
}
s { serial $pdf1 $pdf2 }
m { multiprocess $pdf1 $pdf2 }
p { thread_pool $pdf1 $pdf2 }
t { threaded $pdf1 $pdf2 }
P { promised $pdf1 $pdf2 }
}
}

proc serial {pdf1 pdf2} {
puts -nonewline "serial "
set tsv1 [exec $::PDFTOTEXT $::OPT $pdf1 - 2>@1]
set tsv2 [exec $::PDFTOTEXT $::OPT $pdf2 - 2>@1]
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

proc multiprocess {pdf1 pdf2} {
puts -nonewline multiprocess
set p1 [open "|$::PDFTOTEXT $::OPT $pdf1 - 2>@1" r]
try {
set p2 [open "|$::PDFTOTEXT $::OPT $pdf2 - 2>@1" r]
try {
fconfigure $p1 -blocking 0
fconfigure $p2 -blocking 0
set tsv1 ""
set tsv2 ""
while {![eof $p1] || ![eof $p2]} {
append tsv1 [read $p1]
append tsv2 [read $p2]
after 1
}
} finally {
close $p2
}
} finally {
close $p1
}
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

proc thread_pool {pdf1 pdf2} {
puts -nonewline "thread pool "
set pool [tpool::create -minworkers 2]
set job1 [tpool::post -nowait $pool \
"exec $::PDFTOTEXT $::OPT $pdf1 - 2>@1"]
set job2 [tpool::post -nowait $pool \
"exec $::PDFTOTEXT $::OPT $pdf2 - 2>@1"]
set job_ids [list $job1 $job2]
while {[llength $job_ids] > 0} {
foreach job_id [tpool::wait $pool $job_ids job_ids] {
if {$job_id eq $job1} {
set tsv1 [tpool::get $pool $job_id]
} else {
set tsv2 [tpool::get $pool $job_id]
}
}
}
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

proc threaded {pdf1 pdf2} {
puts -nonewline "threaded "
set tid1 [thread::create -joinable]
set tid2 [thread::create -joinable]
tsv::set shared pdf1 $pdf1
tsv::set shared pdf2 $pdf2
tsv::set shared pdftotext $::PDFTOTEXT
tsv::set shared opt $::OPT
thread::send -async $tid1 {
tsv::set shared tsv1 \
[exec -encoding utf-8 {*}[tsv::get shared pdftotext] \
[tsv::get shared opt] [tsv::get shared pdf1] - 2>@1]
}
thread::send -async $tid2 {
tsv::set shared tsv2 \
[exec -encoding utf-8 {*}[tsv::get shared pdftotext] \
[tsv::get shared opt] [tsv::get shared pdf2] - 2>@1]
}
thread::release $tid1
thread::join $tid1
thread::release $tid2
thread::join $tid2
set tsv1 [tsv::get shared tsv1]
set tsv2 [tsv::get shared tsv2]
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

proc promised {pdf1 pdf2} {
puts -nonewline "promised "
set p1 [promise::pexec $::PDFTOTEXT $::OPT $pdf1 - 2>@1]
set p2 [promise::pexec $::PDFTOTEXT $::OPT $pdf2 - 2>@1]
set waiter [promise::all [list $p1 $p2]]
# Assumes eventloop not running!
lassign [promise::eventloop $waiter] tsv1 tsv2
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

main
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

* saito <saitology9@gmail.com>
| On 5/21/2026 8:21 AM, Ralf Fassel wrote:
| > I bet there is some significant difference between bench.tcl and
| > concurrent.tcl which explains the huge difference between the 'threaded'
| > and 'promised' versions of Ashok and Mark.
| >

| Yes, the difference is significant. I suspect the difference is either
| in pdftotext itself or how it is spawned:

| - It is called with different arguments: "pdftotext pdf1.pdf"
| vs. pdftotext pdf1.pdf pdf1.txt" so perhaps pdftotext behaves
| differently internally to handle output to stdout or to a file.

No, I don't mean the difference betwenn Ashoks and Marks results (this
is to be expected because of different hardware), but why Marks
'promised' version is so much slower than his 'threaded' version (since 'promised' is only a threaded in disguise). Both execute the same
pdftotext program, so this should be the same overhead in both cases.
I'll try Marks version soon.

R'
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On Fri, 22 May 2026 11:34:44 +0200, Ralf Fassel wrote:

* saito <saitology9@gmail.com>
| On 5/21/2026 8:21 AM, Ralf Fassel wrote:
| > I bet there is some significant difference between bench.tcl and
| > concurrent.tcl which explains the huge difference between the 'threaded' | > and 'promised' versions of Ashok and Mark.
| >

| Yes, the difference is significant. I suspect the difference is either
| in pdftotext itself or how it is spawned:

| - It is called with different arguments: "pdftotext pdf1.pdf"
| vs. pdftotext pdf1.pdf pdf1.txt" so perhaps pdftotext behaves
| differently internally to handle output to stdout or to a file.

No, I don't mean the difference betwenn Ashoks and Marks results (this
is to be expected because of different hardware), but why Marks
'promised' version is so much slower than his 'threaded' version (since 'promised' is only a threaded in disguise). Both execute the same
pdftotext program, so this should be the same overhead in both cases.
I'll try Marks version soon.

R'

It is possible that the difference is in different versions of pdftotext.
For example, Ashok said his didn't support the -tsv option and mine does.

pdftotext version 22.12.0
Copyright 2005-2022 The Poppler Developers - http://poppler.freedesktop.org Copyright 1996-2011, 2022 Glyph & Cog, LLC
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

* Mark Summerfield <m.n.summerfield@gmail.com>
| On Thu, 21 May 2026 14:21:18 +0200, Ralf Fassel wrote:
| > I bet there is some significant difference between bench.tcl and
| > concurrent.tcl which explains the huge difference between the 'threaded'
| > and 'promised' versions of Ashok and Mark.
| >
| > Could both of you post the current version of your scripts,
| > so one could retry?
| >
| Sure, below is concurrent.tcl.

| I ran my tests using two very similar PDFs both 647 pages long,
| one 6.6MB the other 6.5MB. For copyright reasons I can't share them.

With 6.6MB PDF size, the overhead on my machine to start the programs is
way longer than the processing time, so I used a 274023565Byte PDF :-) (replicated a PDF 10 times into one).

I slightly modified your code to show the last char of the returned text
- it turns out some versions return the trailing newline while others do
not - but this should not be a problem.

Here are my timings (tcl 9.0.1 as distributed by Debian-13):

=== s ====
serial tsv1=48934{35} tsv2=48934{35}
real 0m0.150s
serial tsv1=48934{35} tsv2=48934{35}
real 0m0.151s
serial tsv1=48934{35} tsv2=48934{35}
real 0m0.152s
serial tsv1=48934{35} tsv2=48934{35}
real 0m0.143s

=== m ====
multiprocess tsv1=48935{10} tsv2=48935{10}
real 0m0.105s
multiprocess tsv1=48935{10} tsv2=48935{10}
real 0m0.093s
multiprocess tsv1=48935{10} tsv2=48935{10}
real 0m0.087s
multiprocess tsv1=48935{10} tsv2=48935{10}
real 0m0.099s

=== p ====
thread pool tsv1=48934{35} tsv2=48934{35}
real 0m0.104s
thread pool tsv1=48934{35} tsv2=48934{35}
real 0m0.085s
thread pool tsv1=48934{35} tsv2=48934{35}
real 0m0.094s
thread pool tsv1=48934{35} tsv2=48934{35}
real 0m0.090s

=== t ====
threaded tsv1=48934{35} tsv2=48934{35}
real 0m0.109s
threaded tsv1=48934{35} tsv2=48934{35}
real 0m0.093s
threaded tsv1=48934{35} tsv2=48934{35}
real 0m0.099s
threaded tsv1=48934{35} tsv2=48934{35}
real 0m0.093s

=== P ====
promised tsv1=48935{10} tsv2=48935{10}
real 0m0.078s
promised tsv1=48935{10} tsv2=48935{10}
real 0m0.082s
promised tsv1=48935{10} tsv2=48935{10}
real 0m0.085s
promised tsv1=48935{10} tsv2=48935{10}
real 0m0.090s

Everything as expected, no difference between Promised and the Threaded
or Multiprocess version, slightly higher execution time for Serial.

So it really puzzles me why you see such a huge timing difference in the 'Promised' version...

R'
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

* Mark Summerfield <m.n.summerfield@gmail.com>
| On Fri, 22 May 2026 11:34:44 +0200, Ralf Fassel wrote:

| > * saito <saitology9@gmail.com>
| > | On 5/21/2026 8:21 AM, Ralf Fassel wrote:
| > | > I bet there is some significant difference between bench.tcl and
| > | > concurrent.tcl which explains the huge difference between the 'threaded'
| > | > and 'promised' versions of Ashok and Mark.
| > | >
| >>
| > | Yes, the difference is significant. I suspect the difference is either
| > | in pdftotext itself or how it is spawned:
| >>
| > | - It is called with different arguments: "pdftotext pdf1.pdf"
| > | vs. pdftotext pdf1.pdf pdf1.txt" so perhaps pdftotext behaves
| > | differently internally to handle output to stdout or to a file.
| >
| > No, I don't mean the difference betwenn Ashoks and Marks results (this
| > is to be expected because of different hardware), but why Marks
| > 'promised' version is so much slower than his 'threaded' version (since
| > 'promised' is only a threaded in disguise). Both execute the same
| > pdftotext program, so this should be the same overhead in both cases.
| > I'll try Marks version soon.
| >
| > R'

| It is possible that the difference is in different versions of pdftotext.
| For example, Ashok said his didn't support the -tsv option and mine does.

Again: it is the difference between the Threaded and Pronmised versions
in *your* report:

1.067 threaded
8.366 promised

where you use (hopefully?) the *same* pdftotext which puzzles me.
The timings should be identical for these two.

With your code, I don't see that difference - see my other post
Message-ID: <yga5x4fohy5.fsf@akutech.de>.

R'
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On Fri, 22 May 2026 12:35:22 +0200, Ralf Fassel wrote:

* Mark Summerfield <m.n.summerfield@gmail.com>
| On Fri, 22 May 2026 11:34:44 +0200, Ralf Fassel wrote:

| > * saito <saitology9@gmail.com>
| > | On 5/21/2026 8:21 AM, Ralf Fassel wrote:
| > | > I bet there is some significant difference between bench.tcl and
| > | > concurrent.tcl which explains the huge difference between the 'threaded'
| > | > and 'promised' versions of Ashok and Mark.
| > | >
| >>
| > | Yes, the difference is significant. I suspect the difference is either | > | in pdftotext itself or how it is spawned:
| >>
| > | - It is called with different arguments: "pdftotext pdf1.pdf"
| > | vs. pdftotext pdf1.pdf pdf1.txt" so perhaps pdftotext behaves
| > | differently internally to handle output to stdout or to a file.
| >
| > No, I don't mean the difference betwenn Ashoks and Marks results (this
| > is to be expected because of different hardware), but why Marks
| > 'promised' version is so much slower than his 'threaded' version (since
| > 'promised' is only a threaded in disguise). Both execute the same
| > pdftotext program, so this should be the same overhead in both cases.
| > I'll try Marks version soon.
| >
| > R'

| It is possible that the difference is in different versions of pdftotext.
| For example, Ashok said his didn't support the -tsv option and mine does.

Again: it is the difference between the Threaded and Pronmised versions
in *your* report:

1.067 threaded
8.366 promised

where you use (hopefully?) the *same* pdftotext which puzzles me.
The timings should be identical for these two.

With your code, I don't see that difference - see my other post
Message-ID: <yga5x4fohy5.fsf@akutech.de>.

R'

Sorry, you're quite right I use only the system pdftotext in all cases.

--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On Fri, 22 May 2026 12:29:54 +0200, Ralf Fassel wrote:

[snip]

Everything as expected, no difference between Promised and the Threaded
or Multiprocess version, slightly higher execution time for Serial.

So it really puzzles me why you see such a huge timing difference in the 'Promised' version...

R'

I certainly can't explain it!

I redid the timings using -bbox instead of -tsv and the disparity was
even bigger on both my desktop & laptop systems:

Desktop computer Tcl/Tk 9.0.3 (64-bit), Debian GNU/Linux 12 (bookworm),
Linux 6.1.0-48-amd64 (x86_64), i7-12700:

$ ./concurrent-run.sh
serial tsv1=22912877 tsv2=22925179 real 1.95
user 1.88
sys 0.15
Method s, Run 1
serial tsv1=22912877 tsv2=22925179 real 1.93
user 1.89
sys 0.12
Method s, Run 2
serial tsv1=22912877 tsv2=22925179 real 1.92
user 1.88
sys 0.12
Method s, Run 3
serial tsv1=22912877 tsv2=22925179 real 1.93
user 1.90
sys 0.12
Method s, Run 4
serial tsv1=22912877 tsv2=22925179 real 1.93
user 1.92
sys 0.09
Method s, Run 5
---------------------------------------------------------
multiprocess tsv1=22912878 tsv2=22925180 real 0.99
user 1.92
sys 0.09
Method m, Run 1
multiprocess tsv1=22912878 tsv2=22925180 real 1.00
user 1.91
sys 0.10
Method m, Run 2
multiprocess tsv1=22912878 tsv2=22925180 real 0.99
user 1.88
sys 0.12
Method m, Run 3
multiprocess tsv1=22912878 tsv2=22925180 real 0.99
user 1.93
sys 0.08
Method m, Run 4
multiprocess tsv1=22912878 tsv2=22925180 real 0.99
user 1.92
sys 0.08
Method m, Run 5
---------------------------------------------------------
thread pool tsv1=22912877 tsv2=22925179 real 1.02
user 1.96
sys 0.11
Method p, Run 1
thread pool tsv1=22912877 tsv2=22925179 real 1.02
user 1.98
sys 0.09
Method p, Run 2
thread pool tsv1=22912877 tsv2=22925179 real 1.02
user 2.01
sys 0.06
Method p, Run 3
thread pool tsv1=22912877 tsv2=22925179 real 1.02
user 1.95
sys 0.12
Method p, Run 4
thread pool tsv1=22912877 tsv2=22925179 real 1.00
user 1.94
sys 0.12
Method p, Run 5
---------------------------------------------------------
threaded tsv1=22912877 tsv2=22925179 real 1.02
user 1.95
sys 0.12
Method t, Run 1
threaded tsv1=22912877 tsv2=22925179 real 1.02
user 1.96
sys 0.10
Method t, Run 2
threaded tsv1=22912877 tsv2=22925179 real 1.02
user 1.97
sys 0.09
Method t, Run 3
threaded tsv1=22912877 tsv2=22925179 real 1.01
user 1.94
sys 0.12
Method t, Run 4
threaded tsv1=22912877 tsv2=22925179 real 1.02
user 1.96
sys 0.11
Method t, Run 5
---------------------------------------------------------
promised tsv1=22912878 tsv2=22925180 real 16.28
user 18.10
sys 0.28
Method P, Run 1
promised tsv1=22912878 tsv2=22925180 real 16.32
user 18.17
sys 0.25
Method P, Run 2
promised tsv1=22912878 tsv2=22925180 real 16.28
user 18.06
sys 0.32
Method P, Run 3
promised tsv1=22912878 tsv2=22925180 real 16.27
user 18.00
sys 0.37
Method P, Run 4
promised tsv1=22912878 tsv2=22925180 real 16.32
user 18.04
sys 0.39
Method P, Run 5
---------------------------------------------------------

Laptop computer Tcl/Tk 9.0.3 (64-bit), Ubuntu 22.04.5 LTS,
Linux 6.8.0-117-generic (x86_64), i5-6200U

$ ./concurrent-run.sh
serial tsv1=22912799 tsv2=22925105 real 6.12
user 5.92
sys 0.46
Method s, Run 1
serial tsv1=22912799 tsv2=22925105 real 5.90
user 5.77
sys 0.46
Method s, Run 2
serial tsv1=22912799 tsv2=22925105 real 5.86
user 5.76
sys 0.45
Method s, Run 3
serial tsv1=22912799 tsv2=22925105 real 5.73
user 5.60
sys 0.47
Method s, Run 4
serial tsv1=22912799 tsv2=22925105 real 5.71
user 5.59
sys 0.47
Method s, Run 5
---------------------------------------------------------
multiprocess tsv1=22912800 tsv2=22925106 real 3.10
user 5.90
sys 0.47
Method m, Run 1
multiprocess tsv1=22912800 tsv2=22925106 real 3.04
user 5.75
sys 0.44
Method m, Run 2
multiprocess tsv1=22912800 tsv2=22925106 real 2.96
user 5.70
sys 0.45
Method m, Run 3
multiprocess tsv1=22912800 tsv2=22925106 real 2.96
user 5.68
sys 0.47
Method m, Run 4
multiprocess tsv1=22912800 tsv2=22925106 real 3.01
user 5.74
sys 0.41
Method m, Run 5
---------------------------------------------------------
thread pool tsv1=22912799 tsv2=22925105 real 3.35
user 6.24
sys 0.52
Method p, Run 1
thread pool tsv1=22912799 tsv2=22925105 real 3.33
user 6.40
sys 0.52
Method p, Run 2
thread pool tsv1=22912799 tsv2=22925105 real 3.31
user 6.37
sys 0.48
Method p, Run 3
thread pool tsv1=22912799 tsv2=22925105 real 3.36
user 6.38
sys 0.49
Method p, Run 4
thread pool tsv1=22912799 tsv2=22925105 real 3.32
user 6.31
sys 0.47
Method p, Run 5
---------------------------------------------------------
threaded tsv1=22912799 tsv2=22925105 real 3.35
user 6.39
sys 0.51
Method t, Run 1
threaded tsv1=22912799 tsv2=22925105 real 3.36
user 6.37
sys 0.55
Method t, Run 2
threaded tsv1=22912799 tsv2=22925105 real 4.16
user 7.54
sys 0.59
Method t, Run 3
threaded tsv1=22912799 tsv2=22925105 real 3.34
user 6.37
sys 0.50
Method t, Run 4
threaded tsv1=22912799 tsv2=22925105 real 3.38
user 6.40
sys 0.49
Method t, Run 5
---------------------------------------------------------
promised tsv1=22912800 tsv2=22925106 real 35.24
user 39.67
sys 1.53
Method P, Run 1
promised tsv1=22912800 tsv2=22925106 real 35.17
user 39.51
sys 1.55
Method P, Run 2
promised tsv1=22912800 tsv2=22925106 real 35.74
user 40.02
sys 1.68
Method P, Run 3
promised tsv1=22912800 tsv2=22925106 real 35.44
user 39.80
sys 1.64
Method P, Run 4
promised tsv1=22912800 tsv2=22925106 real 35.25
user 40.50
sys 1.22
Method P, Run 5
---------------------------------------------------------
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On Thu, 21 May 2026 16:03:33 +0530, Ashok wrote:

On 5/18/2026 12:41 PM, Mark Summerfield wrote:

The most obvious difference is that I am running Linux on the hardware
and (I think) you are running Linux on Windows. All I can suggest is
trying the same test on Linux that's running directly on the hardware?

Don't have such a beast :-)

[snip]

If you had the time you could though. It would mean creating a
Linux "live" DVD image on a USB memory stick and booting from that.
Then you'd need to download into the live Linux your timing shell
script (wish sh changed to bash most likely), bench.tcl, a Tcl 9
(e.g., one of the binaries like freeWrap), and a couple of big PDFs
and you'd have a real test directly on your hardware. This won't
touch your HDD or SDD (so long as you don't click Install:-)
At the end just shutdown Linux and reboot and your machine is
exactly as it was.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On 5/22/2026 4:13 AM, Mark Summerfield wrote:

On Fri, 22 May 2026 12:35:22 +0200, Ralf Fassel wrote:

* Mark Summerfield <m.n.summerfield@gmail.com>
| On Fri, 22 May 2026 11:34:44 +0200, Ralf Fassel wrote:

| > * saito <saitology9@gmail.com>
| > | On 5/21/2026 8:21 AM, Ralf Fassel wrote:
| > | > I bet there is some significant difference between bench.tcl and
| > | > concurrent.tcl which explains the huge difference between the 'threaded'
| > | > and 'promised' versions of Ashok and Mark.
| > | >
| >>
| > | Yes, the difference is significant. I suspect the difference is either >> | > | in pdftotext itself or how it is spawned:
| >>
| > | - It is called with different arguments: "pdftotext pdf1.pdf"
| > | vs. pdftotext pdf1.pdf pdf1.txt" so perhaps pdftotext behaves
| > | differently internally to handle output to stdout or to a file.
| >
| > No, I don't mean the difference betwenn Ashoks and Marks results (this >> | > is to be expected because of different hardware), but why Marks
| > 'promised' version is so much slower than his 'threaded' version (since >> | > 'promised' is only a threaded in disguise). Both execute the same
| > pdftotext program, so this should be the same overhead in both cases.
| > I'll try Marks version soon.
| >
| > R'

| It is possible that the difference is in different versions of pdftotext. >> | For example, Ashok said his didn't support the -tsv option and mine does. >>
Again: it is the difference between the Threaded and Pronmised versions
in *your* report:

1.067 threaded
8.366 promised

where you use (hopefully?) the *same* pdftotext which puzzles me.
The timings should be identical for these two.

With your code, I don't see that difference - see my other post
Message-ID: <yga5x4fohy5.fsf@akutech.de>.

R'

Sorry, you're quite right I use only the system pdftotext in all cases.

I was able to reproduce the timings where promises was using much more real time. My setup is a vmware ubuntu running on windows 10 on a 4 core machine.

If I ran with a smallish pdf file pair which reported about 1mb output size of the text, the timings were as expected. When I ran with a very large pdf pair, and the output was 78mb, promises would slow down dramatically.

In addition, with top running in a window, when promises ran with the large file, it would show that tclsh was pegged at 100% with each of the two pdftotext only about 10%

When running the several versions of threading, it would show the two pdftotext at 100% with tclsh at 5%

There was one anomaly, where one time a tpool took as long as serial, and when that ran, only 1 pdftotext was at 100%.

One other data point, my VM only has 3.5 gig of ram so I likely was running out and causing pagefaults. Perhaps that's what Mark was seeing. When promises was running on the big data set, monitoring vmware in windows showed that vmware was only using 1 cpu, not 2 as in other cases. That too points to paging issues I would think.

Therefore I would look to memory issues in promises causing it to run slowly on large data sets. Perhaps it uses a lot more memory than the thread/tpool/multiproc versions, but perhaps one has to be short on allocated ram and causing page faults to see the problem.

-e

--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

Am 24.05.26 um 07:06 schrieb et99:

In addition, with top running in a window, when promises ran with the
large file, it would show that tclsh was pegged at 100% with each of the
two pdftotext only about 10%

What about encoding conversion? In some of Marks code there is
"-encoding utf8". Also, it may be different if the pdftotext spits out
every single character as an event vs. larger chunks of data. Would be interesting to count the number of events for each calls.

Christian
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

* et99 <et99@rocketship1.me>
--<snip-snip>--
| I was able to reproduce the timings where promises was using much more
| real time. My setup is a vmware ubuntu running on windows 10 on a 4
| core machine.

| If I ran with a smallish pdf file pair which reported about 1mb output
| size of the text, the timings were as expected. When I ran with a very
| large pdf pair, and the output was 78mb, promises would slow down
| dramatically.

| In addition, with top running in a window, when promises ran with the
| large file, it would show that tclsh was pegged at 100% with each of
| the two pdftotext only about 10%

| When running the several versions of threading, it would show the two
| pdftotext at 100% with tclsh at 5%

Ok, now we're getting somewhere. With 75MB of text output, I see the
same as et99: s/m/t/p as expected (pdftotext 2x 100%CPU, tclsh waiting
for the result), but P has tclsh at 100% CPU and both pdftotext at 10%.

Looking at the promise 'pexec' code, I see that I was wrong in assuming
it is "threads-in-disguise". Instead it sets up two pipes and reads
them until EOF, causing many reallocations of the huge strings. Somehow
this is not as effective as 'exec' itself, which also needs to read from
the pipes, but obviously does it more efficiently.

Perhaps using promise::ptask or promise::pworker would give better
results.

R'
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

Thanks for the analysis, all. I just haven't had time to follow up on
this regularly.

As to the explanation,

- the multi-process read is more efficient because it blocks for 1
second so reads larger chunks of data with fewer appends. pexec could be modified similarly. The drawback of this approach, besides blocking for
1 second, is the potential of the forced 1 second latencies (exec's that actually finished in 1.01 secs would take 2 seconds, 2.01->3 secs etc.).

- multi-thread execs read in one big chunk. ptask or pthread would give similar results, I think as Ralf suggested though I have not actually
tried it.

I'll take a further look at some later point when I have some time about optimizing pexec, but not clear how since I'm not in favor of blocking
or polling as in the multi-process case. Perhaps in 9.0, the new process commands will allow waiting till the process completes instead of
reading via events.


/Ashok


On 5/24/2026 8:54 PM, Ralf Fassel wrote:

* et99 <et99@rocketship1.me>
--<snip-snip>--
| I was able to reproduce the timings where promises was using much more
| real time. My setup is a vmware ubuntu running on windows 10 on a 4
| core machine.

| If I ran with a smallish pdf file pair which reported about 1mb output
| size of the text, the timings were as expected. When I ran with a very
| large pdf pair, and the output was 78mb, promises would slow down
| dramatically.

| In addition, with top running in a window, when promises ran with the
| large file, it would show that tclsh was pegged at 100% with each of
| the two pdftotext only about 10%

| When running the several versions of threading, it would show the two
| pdftotext at 100% with tclsh at 5%

Ok, now we're getting somewhere. With 75MB of text output, I see the
same as et99: s/m/t/p as expected (pdftotext 2x 100%CPU, tclsh waiting
for the result), but P has tclsh at 100% CPU and both pdftotext at 10%.

Looking at the promise 'pexec' code, I see that I was wrong in assuming
it is "threads-in-disguise". Instead it sets up two pipes and reads
them until EOF, causing many reallocations of the huge strings. Somehow
this is not as effective as 'exec' itself, which also needs to read from
the pipes, but obviously does it more efficiently.

Perhaps using promise::ptask or promise::pworker would give better
results.

R'

--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

Am 26.05.26 um 05:26 schrieb Ashok:

Thanks for the analysis, all. I just haven't had time to follow up on
this regularly.

As to the explanation,

- the multi-process read is more efficient because it blocks for 1
second so reads larger chunks of data with fewer appends.

I don't get it. "after 1" waits for *1 Millisecond*. 1 second would be
"after 1000".

Christian


--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

The multi-process blocks for 1 millisecond not 1 second (the after
command takes milliseconds).

Regarding RAM, my desktop system has 16GB and concurrent.tcl uses
a couple of percent of it (in all cases), so for my timings
there's no swapping.


On Tue, 26 May 2026 08:56:46 +0530, Ashok wrote:

Thanks for the analysis, all. I just haven't had time to follow up on
this regularly.

As to the explanation,

- the multi-process read is more efficient because it blocks for 1
second so reads larger chunks of data with fewer appends. pexec could be modified similarly. The drawback of this approach, besides blocking for
1 second, is the potential of the forced 1 second latencies (exec's that actually finished in 1.01 secs would take 2 seconds, 2.01->3 secs etc.).

- multi-thread execs read in one big chunk. ptask or pthread would give similar results, I think as Ralf suggested though I have not actually
tried it.

I'll take a further look at some later point when I have some time about optimizing pexec, but not clear how since I'm not in favor of blocking
or polling as in the multi-process case. Perhaps in 9.0, the new process commands will allow waiting till the process completes instead of
reading via events.

/Ashok

[snip]
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

* Ashok <apnmbx-public@yahoo.com>
| Thanks for the analysis, all. I just haven't had time to follow up on
| this regularly.

| As to the explanation,

| - the multi-process read is more efficient because it blocks for 1
| second so reads larger chunks of data with fewer appends. pexec
| could be modified similarly.

As others have pointed out the wait is 1 *millisecond* (which still is
aeons in computer parlance), not 1 second.

However, with 26MB of text output, if I look at the data chunks as they
are read (simply printing out the string length of each chunk to stderr
and redirect stderr to a file), I see that the 'promise' code gets
chunks of 65k in each read, while the 'multiprocess' gets only 8k.

So in contrast the multiprocess reads *fewer* bytes in each [read],
while the pexec reads *more*.

Still multiprocess takes only 3.7s to complete:
[...]
file5 appending 8192 bytes
file6 appending 8192 bytes
file5 appending 12288 bytes
file6 appending 8192 bytes
file5 appending 8192 bytes
file6 appending 8192 bytes
file5 appending 8192 bytes
file6 appending 8192 bytes
file5 appending 8192 bytes
file6 appending 8192 bytes
file5 appending 8192 bytes
file6 appending 8192 bytes
file5 appending 8192 bytes
file6 appending 8192 bytes
file5 appending 8192 bytes
file6 appending 12288 bytes
--<snip-snip>--
file6 appending 8192 bytes
file5 appending 0 bytes
file6 appending 12288 bytes
file5 appending 0 bytes
file6 appending 8192 bytes
file5 appending 0 bytes
file6 appending 8192 bytes
file5 appending 0 bytes
file6 appending 0 bytes
file5 appending 0 bytes
file6 appending 0 bytes
file5 appending 0 bytes
file6 appending 0 bytes
file5 appending 0 bytes
file6 appending 0 bytes
file5 appending 0 bytes
file6 appending 689 bytes
file5 appending 0 bytes
file6 appending 0 bytes
file5 appending 0 bytes
file6 appending 0 bytes
file5 appending 0 bytes
file6 appending 0 bytes
real 3,67
user 6,59
sys 0,88


while promise::pexec takes about 20s
[...]
file5 appending 69632 bytes
file6 appending 69632 bytes
file5 appending 69632 bytes
file6 appending 69632 bytes
file5 appending 69632 bytes
file6 appending 69632 bytes
file5 appending 69632 bytes
file6 appending 69632 bytes
file5 appending 45745 bytes
file6 appending 17073 bytes
real 20,42
user 26,44
sys 1,30

I verified that the sum of bytes-read in each case is as expected.
Very confusing.

R'
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

* Ashok <apnmbx-public@yahoo.com>
| - multi-thread execs read in one big chunk. ptask or pthread would
| give similar results, I think as Ralf suggested though I have not
| actually tried it.

I have added

proc ptask {pdf1 pdf2} {
puts -nonewline "ptask "
set p1 [promise::ptask [list exec -keepnewline $::PDFTOTEXT $::OPT $pdf1 - 2>@1]]
set p2 [promise::ptask [list exec -keepnewline $::PDFTOTEXT $::OPT $pdf2 - 2>@1]]
set waiter [promise::all [list $p1 $p2]]
# Assumes eventloop not running!
lassign [promise::eventloop $waiter] tsv1 tsv2
puts " tsv1=[string length $tsv1] tsv2=[string length $tsv2]"
}

to Mark's code and get results as expected ('T' being the 'ptask' result):

=== m ====
real 3,63
user 6,65
sys 0,71
multiprocess tsv1=26305201 tsv2=26305201

=== T ====
real 3,84
user 7,02
sys 0,71
ptask tsv1=26305201 tsv2=26305201

While still pexec is way slower
=== P ====
real 20,47
user 26,66
sys 1,25
promised tsv1=26305201 tsv2=26305201

R'
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

* Ashok <apnmbx-public@yahoo.com>
| I'll take a further look at some later point when I have some time
| about optimizing pexec, but not clear how since I'm not in favor of
| blocking or polling as in the multi-process case. Perhaps in 9.0, the
| new process commands will allow waiting till the process completes
| instead of reading via events.

Ok, I think I have found the explanation for this:

in proc promise::_read_channel, you reset the fileevent after each
read with the new data, like this:

fileevent $chan readable [list [namespace current]::_read_channel $prom $chan $data]

this way the data gets transported without the need of a 'storage'
variable.


However, Tcl_FileEventObjCmd() in generic/tclIO.c contains this section
of code:

/*
* If we are supposed to delete a stored script, do so.
*/

if (*(TclGetString(objv[3])) == '\0') {
DeleteScriptRecord(interp, chanPtr, mask);
return TCL_OK;
}

I.e. the string rep of the fileevent script is checked to be empty, and
this probably is what causes the slowdown (shimmering from list to
string after each read).

If I BFI use a global variable to append the strings and *not* reset the fileevent:

proc promise::_read_channel {prom chan data} {
set newdata [read $chan]
if {[string length $newdata] || ![eof $chan]} {
# BFI use global for testing purpose
append ::mydata($chan) $newdata
# do not reset the fileevent
# fileevent $chan readable [list [namespace current]::_read_channel $prom $chan $data]
return
}
...
safe_fulfill $prom $::mydata($chan)
}

Then the pexec performance is similar to the others:

=== t ====
real 3,69
user 6,72
sys 0,74
threaded tsv1=26305200 tsv2=26305200

=== P ====
real 3,67
user 6,73
sys 0,85
promised tsv1=26305201 tsv2=26305201

HTH
R'
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

* Ralf Fassel <ralfixx@gmx.de>
| However, Tcl_FileEventObjCmd() in generic/tclIO.c contains this section
| of code:

| /*
| * If we are supposed to delete a stored script, do so.
| */

| if (*(TclGetString(objv[3])) == '\0') {
| DeleteScriptRecord(interp, chanPtr, mask);
| return TCL_OK;
| }

| I.e. the string rep of the fileevent script is checked to be empty, and
| this probably is what causes the slowdown (shimmering from list to
| string after each read).

See https://core.tcl-lang.org/tcl/tktview?name=7da6c2d04c219e8e

R'
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

Am 26.05.2026 um 19:59 schrieb Ralf Fassel:

* Ralf Fassel <ralfixx@gmx.de>
| However, Tcl_FileEventObjCmd() in generic/tclIO.c contains this section
| of code:

| /*
| * If we are supposed to delete a stored script, do so.
| */

| if (*(TclGetString(objv[3])) == '\0') {
| DeleteScriptRecord(interp, chanPtr, mask);
| return TCL_OK;
| }

| I.e. the string rep of the fileevent script is checked to be empty, and
| this probably is what causes the slowdown (shimmering from list to
| string after each read).

See https://core.tcl-lang.org/tcl/tktview?name=7da6c2d04c219e8e

R'

Wow, Ralf, Wizard level analysis! We are all impressed!
Cudos,
Harald
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

Am 27.05.2026 um 10:15 schrieb Harald Oehlmann:

Am 26.05.2026 um 19:59 schrieb Ralf Fassel:

* Ralf Fassel <ralfixx@gmx.de>
| However, Tcl_FileEventObjCmd() in generic/tclIO.c contains this section
| of code:

|     /*
|      * If we are supposed to delete a stored script, do so.
|      */

|     if (*(TclGetString(objv[3])) == '\0') {
|     DeleteScriptRecord(interp, chanPtr, mask);
|     return TCL_OK;
|     }

| I.e. the string rep of the fileevent script is checked to be empty, and
| this probably is what causes the slowdown (shimmering from list to
| string after each read).

See https://core.tcl-lang.org/tcl/tktview?name=7da6c2d04c219e8e

R'

Wow, Ralf, Wizard level analysis! We are all impressed!
Cudos,
Harald

The thread lead to this Tk ticket: https://core.tcl-lang.org/tcl/info/7da6c2d04c
with a set of solutions.

Thanks for all,
Harald
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On 5/28/2026 4:49 AM, Harald Oehlmann wrote:

The thread lead to this Tk ticket: https://core.tcl-lang.org/tcl/info/7da6c2d04c
with a set of solutions.

Thanks for all,
Harald

Congrats to for the fix.

It says that the pattern is quite common. I read it to mean the pattern
of re-creating the event handler, within the event handler itself, with
all the data read so far.

Is this really a common pattern? Wouldn't this assume the programmer is relying on an unlimited amount of memory being available to handle large files? How about a streaming channel where the data keeps coming?

So perhaps there was no need for concern since the programmer can easily rectify it?








--- Synchronet 3.22a-Linux NewsLink 1.2