From Newsgroup: comp.sys.ibm.pc.hardware.storage

I have a folder on one of my SSD drives that takes 8 to 10 hours to back
up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.

Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this inefficient for small files like this?

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:

I have a folder on one of my SSD drives that takes 8 to 10 hours to back
up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.

Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this inefficient for small files like this?

Using WHAT backup software? Doing a file-based or image-based backup?

Is it a direct access to the folder, or are you using a redirection,
like a junction (reparse point)? Does that folder itself have any
redirections which could run the backup program into a loop if it
doesn't specifically ignore those?
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan wrote:

I have a folder on one of my SSD drives that takes 8 to 10 hours to back
up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.

Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this inefficient for small files like this?

Yousuf Khan

Have you tried to "defragment" the drive ?

Normally, the "optimize" dialog will not offer defragmentation
as an option in Windows 10. It's supposed to offer "TRIM" as
the option for an SSD.

However, there is a "Copy On Write" or COW issue with SSDs.
Under the right circumstances, there will be a slowdown.

Now, consider what you're doing. Your backup software uses VSS
to make a shadow copy. It's possible some "COW activity" is happening
during the backup.

The Optimize dialog knows about this, and the Optimize dialog
has some sort of metric it uses to decide what to do. While
most of the time, it will only offer TRIM, I bet in your
case, it's "going to have a COW" and defragment your drive.
This should not be as thorough as a regular defragment,
and the design of what's done, should have something to do
with whatever the root cause of "having a COW" is.

I've not seen this slow behavior here, so have no
first hand experiences to offer on it. Note that over the
years Windows 10 has existed, the behavior of the Optimize
panel has been "as crazy as Cocoa Puffs". The software
frequently could not properly tell an HDD from an SSD,
and it would be damn hard to see any "subtle" behaviors,
when this software has had so many bugs in the past. I've
had a machine full of HDDs offer nothing but TRIM and
the Optimize panel declared all my drives as SSD drives.
Which is total bullshit and most annoying when you
actually want the defrag to work. As far as I can remember,
Optimize is working in 1909 OK now. It's been a hell of a
bumpy ride though, over the years.

See if you're offered a defrag option.

Do Properties on the drive letter, and in the Tools tab
you'll find the Optimize. Then retest your backup rate
after the partition has been cleaned up.

Paul
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

VanguardLH <V@nguard.LH> wrote in news:3j6lwsi2gc3u.dlg@v.nguard.lh:

Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:

I have a folder on one of my SSD drives that takes 8 to 10 hours to
back up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.

Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this
inefficient for small files like this?

Using WHAT backup software? Doing a file-based or image-based backup?

Is it a direct access to the folder, or are you using a redirection,
like a junction (reparse point)? Does that folder itself have any redirections which could run the backup program into a loop if it
doesn't specifically ignore those?

That may be it. I remember reading about junctions causing havoc if they
were in a backup scheme (I think in a folder/file backup). Amazingly,
there was some helpful information (for me, at the time) on a Microsoft
forum, about identifying junctions, found in paragraph two of darrenc1's answer.

To the OP:

https://answers.microsoft.com/en-us/windows/forum/windows_7- performance/what-is-a-reparse-point-can-anyone-reveal-the/17b9b457-6c8a- 4e83-a445-e603011a6b95

or

https://tinyurl.com/y8hssmg6

--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 4/26/2020 9:32 PM, VanguardLH wrote:

Using WHAT backup software? Doing a file-based or image-based backup?

Macrium, file-based.

Is it a direct access to the folder, or are you using a redirection,
like a junction (reparse point)? Does that folder itself have any redirections which could run the backup program into a loop if it
doesn't specifically ignore those?

No, none of that. Straightforward unredirected.

Yousuf Khan

--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 4/26/2020 9:46 PM, Paul wrote:

Have you tried to "defragment" the drive ?

No, considering it's an SSD. But as you pointed out later, the optimize
option is available for both of my SSD's, but optimize recognizes them
as SSD's, so the only optimization available is trimming, no defragging.

Normally, the "optimize" dialog will not offer defragmentation
as an option in Windows 10. It's supposed to offer "TRIM" as
the option for an SSD.

However, there is a "Copy On Write" or COW issue with SSDs.
Under the right circumstances, there will be a slowdown.

Yes, likely this is exactly that circumstance. Do you know what the
symptoms of that circumstance are?

Now, consider what you're doing. Your backup software uses VSS
to make a shadow copy. It's possible some "COW activity" is happening
during the backup.

Yes, VSS is used by the software, which is Macrium Reflect 6 BTW.
Reflect's logs show that it creates the VSS shadows immediately before beginning the backup.

This backup runs after midnight, and there is little activity while any
of the backups run. All of the backups run after midnight and they
finish relatively quickly, except this one.

The Optimize dialog knows about this, and the Optimize dialog
has some sort of metric it uses to decide what to do. While
most of the time, it will only offer TRIM, I bet in your
case, it's "going to have a COW" and defragment your drive.
This should not be as thorough as a regular defragment,
and the design of what's done, should have something to do
with whatever the root cause of "having a COW" is.

VSS is used on all of the backup jobs. None of the others exhibit this behaviour. In fact, I've experienced this issue for nearly a decade now.
The problem started on Windows XP, continued on into Windows 7, and
continues to plague me in Windows 10. This particular folder has also
been migrated around from HDD to SSD, to a 2nd SSD, etc. So it's not a
problem that is specific to HDD's or SSD's, or to any particular version
of Windows.

I'll tell you what this folder is. It's actually my Thunderbird News
folder (exactly what I'm using to ask this question here), which exists
under the my User folder structure. The problem was discovered when I
started doing daily backups of my User folder and discovered that the
User folder was taking forever. After investigating it some, I figured
out that the problem was this particular substructure under News. Once I excluded the News folder, backups finished 6 times faster! So I moved
the backups of the News folder to their own job, and let the rest of the
User folder get backed up separately. Before, you ask, I only backup the
News folder once a week, but it's still a pain in the ass watching it
take so long even once a week.

Some other background. When this particular backup is happening, it's
not the drives that are showing as busy, it's the CPU cores! 4 out of
the 8 cores on my FX-8300 are fluctuating between 50% to 100% busy,
while the other 4 are not that busy.

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan wrote:

Some other background. When this particular backup is happening, it's
not the drives that are showing as busy, it's the CPU cores! 4 out of
the 8 cores on my FX-8300 are fluctuating between 50% to 100% busy,
while the other 4 are not that busy.

Yousuf Khan

I seem to remember at some time in the past, you offered
advice on putting an exception for an AV program,
so it does not scan that particular directory
(something in Thunderbird).

If your CPU cores are railed, I'd be tracing down the
PID of the offender.

One way to do it on a Pro SKU of OS, is

tasklist /svc # should not work on Home

and that will tell you what is inside a SVCHOST. You
can also do that with Process Explorer from Sysinternals,
running concurrently with Task Manager, and flip over
to Process Explorer to see what is in a busy PID in
Task Manager. If you elevate Process Explorer using
"Run as Administrator", it can even take a stack snapshot
of a SVCHOST, and you can get additional information.

For example, I have a SVCHOST with 15 things in it,
and one is wuauserv. If a Windows Update scan is running,
that SVCHOST lights up -- but then you have to guess
that's the guilty service, as the rest of the services
aren't normally a problem.

When Macrium is running, CPU effort goes into two things:

1) Running a checksum to stamp the .mrimg when finished.
This detects corruption later (like when restoring perhaps).

2) Compression. If the lightweight compressor is turned on,
that will use a core. I don't think Macrium uses multi-core
for its compressor.

If you were seeing more than that, I'd be looking at
MsMpEng as a culprit, as it could cause quite a penalty
if every small file involved a scan by the Windows Defender.

When I ran hashdeep64 in Windows 10, I think the calc
ran 8x slower than normal, to give some idea what a
penalty Windows Defender causes on reads.

Paul
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 4/27/2020 2:06 AM, Yousuf Khan wrote:

On 4/26/2020 9:46 PM, Paul wrote:

Have you tried to "defragment" the drive ?

No, considering it's an SSD. But as you pointed out later, the optimize option is available for both of my SSD's, but optimize recognizes them
as SSD's, so the only optimization available is trimming, no defragging.

Normally, the "optimize" dialog will not offer defragmentation
as an option in Windows 10. It's supposed to offer "TRIM" as
the option for an SSD.

However, there is a "Copy On Write" or COW issue with SSDs.
Under the right circumstances, there will be a slowdown.

Yes, likely this is exactly that circumstance. Do you know what the
symptoms of that circumstance are?

Now, consider what you're doing. Your backup software uses VSS
to make a shadow copy. It's possible some "COW activity" is happening
during the backup.

Yes, VSS is used by the software, which is Macrium Reflect 6 BTW.
Reflect's logs show that it creates the VSS shadows immediately before beginning the backup.

This backup runs after midnight, and there is little activity while any
of the backups run. All of the backups run after midnight and they
finish relatively quickly, except this one.

The Optimize dialog knows about this, and the Optimize dialog
has some sort of metric it uses to decide what to do. While
most of the time, it will only offer TRIM, I bet in your
case, it's "going to have a COW" and defragment your drive.
This should not be as thorough as a regular defragment,
and the design of what's done, should have something to do
with whatever the root cause of "having a COW" is.

VSS is used on all of the backup jobs. None of the others exhibit this behaviour. In fact, I've experienced this issue for nearly a decade now.
The problem started on Windows XP, continued on into Windows 7, and continues to plague me in Windows 10. This particular folder has also
been migrated around from HDD to SSD, to a 2nd SSD, etc. So it's not a problem that is specific to HDD's or SSD's, or to any particular version
of Windows.

I'll tell you what this folder is. It's actually my Thunderbird News
folder (exactly what I'm using to ask this question here), which exists under the my User folder structure. The problem was discovered when I started doing daily backups of my User folder and discovered that the
User folder was taking forever. After investigating it some, I figured
out that the problem was this particular substructure under News. Once I excluded the News folder, backups finished 6 times faster! So I moved
the backups of the News folder to their own job, and let the rest of the User folder get backed up separately. Before, you ask, I only backup the News folder once a week, but it's still a pain in the ass watching it
take so long even once a week.

Some other background. When this particular backup is happening, it's
not the drives that are showing as busy, it's the CPU cores! 4 out of
the 8 cores on my FX-8300 are fluctuating between 50% to 100% busy,
while the other 4 are not that busy.

    Yousuf Khan

Not really sure, but I think TB does compression on its files. If you
don't allow it, that might be the cause.
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 4/27/2020 8:28 AM, Todesco wrote:

Not really sure, but I think TB does compression on its files.  If you don't allow it, that might be the cause.

It does that only when it's active and running, in this case it's not
running. Also it doesn't compress newsgroup files, just email files.
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 4/27/2020 3:57 AM, Paul wrote:

I seem to remember at some time in the past, you offered
advice on putting an exception for an AV program,
so it does not scan that particular directory
(something in Thunderbird).

If your CPU cores are railed, I'd be tracing down the
PID of the offender.

One way to do it on a Pro SKU of OS, is

   tasklist /svc                # should not work on Home

Not even necessary, I can tell you right now which process is
responsible, it's the Macrium Reflect binary. Also the System process
which I assume the Reflect binary also makes heavy use of during this time.
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 4/26/2020 6:46 PM, Paul wrote:

Yousuf Khan wrote:

I have a folder on one of my SSD drives that takes 8 to 10 hours to back
up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.

Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this
inefficient for small files like this?

Yousuf Khan

Have you tried to "defragment" the drive ?

Since the folder is on an SSD, fragmentation shouldn't make any difference.

--
Ken
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:
[...]

VSS is used on all of the backup jobs. None of the others exhibit this behaviour. In fact, I've experienced this issue for nearly a decade now.
The problem started on Windows XP, continued on into Windows 7, and continues to plague me in Windows 10. This particular folder has also
been migrated around from HDD to SSD, to a 2nd SSD, etc. So it's not a problem that is specific to HDD's or SSD's, or to any particular version
of Windows.

I'll tell you what this folder is. It's actually my Thunderbird News
folder (exactly what I'm using to ask this question here), which exists under the my User folder structure. The problem was discovered when I started doing daily backups of my User folder and discovered that the
User folder was taking forever. After investigating it some, I figured
out that the problem was this particular substructure under News. Once I excluded the News folder, backups finished 6 times faster! So I moved
the backups of the News folder to their own job, and let the rest of the User folder get backed up separately. Before, you ask, I only backup the News folder once a week, but it's still a pain in the ass watching it
take so long even once a week.

If there are 580,000 files in the News folder, then you've probably configured your Thunderbird News account(s) to use one file for each
article instead of one file for each newsgroup.

If so, it's probably best to bite the bullet and convert to one file
per newsgroup. That probably needs an export and (re-)import and
probably will be time-consuming, but at least then you'll solve the
actual problem.

FYI, my setup - not Thunderbird - has nearly a million articles, but
only some 600 files.

Some other background. When this particular backup is happening, it's
not the drives that are showing as busy, it's the CPU cores! 4 out of
the 8 cores on my FX-8300 are fluctuating between 50% to 100% busy,
whole the other 4 are not that busy.

My guess it that this processing is spent getting the hundreds of
thousands of files into and out of the file system cache.

Yousuf Khan

--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

This is a multi-part message in MIME format. --------------2E47DE48F3351ED3588A7BF8
Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit

On 27/04/2020 02:24, Yousuf Khan wrote:

I have a folder on one of my SSD drives that takes 8 to 10 hours to
back up. It is only about 1.4 GB, but it is allocated 2.4 GB of space altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.

Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this inefficient for small files like this?

NTFS is pretty efficient but some users of Windows 10 machine aren't. 
Also, your machine must be showing signs of suspicious activities so the backup program must scan it to see if there are any imminent threat to
the society in general by your sordid activities.



--
With over 1.2 billion devices now running Windows 10, customer
satisfaction is higher than any previous version of windows.

--------------2E47DE48F3351ED3588A7BF8
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#008000" bgcolor="#faf0e6">
<div class="moz-cite-prefix">On 27/04/2020 02:24, Yousuf Khan wrote:<br>
</div>
<blockquote type="cite"
cite="mid:Mp6dncpR2YPPqTvDnZ2dnUU7-budnZ2d@giganews.com">I have a
folder on one of my SSD drives that takes 8 to 10 hours to back
up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per
file it's using up a little bit over half of a cluster on average.
File system is NTFS.
<br>
<br>
Meanwhile, this same drive can backup the remainder of the drive
in under 2 hours, and the remainder of the drive is 390 GB! Is
NTFS this inefficient for small files like this?
<br>
<br>
   <br>
</blockquote>
<p><br>
</p>
<p>NTFS is pretty efficient but some users of Windows 10 machine
aren't.  Also, your machine must be showing signs of suspicious
activities so the backup program must scan it to see if there are
any imminent threat to the society in general by your sordid
activities.</p>
<p><br>
</p>
<p><br>
</p>
<div class="moz-signature">-- <br>
<div style="width: 340px;height: 290px; background-color: blue;
color: yellow;font-weight: bolder; font-size:200%; text-align:
center; margin: 30px 5px 30px 5px;">With over 1.2 billion
devices now running Windows 10, customer satisfaction is higher
than any previous version of windows.</div>
</div>
</body>
</html>

--------------2E47DE48F3351ED3588A7BF8--
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:

On 4/26/2020 9:32 PM, VanguardLH wrote:

Using WHAT backup software? Doing a file-based or image-based backup?

Macrium, file-based.

Is it a direct access to the folder, or are you using a redirection,
like a junction (reparse point)? Does that folder itself have any
redirections which could run the backup program into a loop if it
doesn't specifically ignore those?

No, none of that. Straightforward unredirected.

What did you use to check if there were junctions defined within the
folder? For example, you could use Nirsoft's NTFSLinksView tool to scan
for junctions to list them. You can specify the start folder from where
to search, like the folder with the 500K+ files, or search from the root
folder of a drive (junctions cannot point to other drives). Alas, if
you pick the problematic folder, a scan will only show any junctions in
that folder, not those that point at that folder. You might want to
scan from the root folder, and then check if that folder is under a
junction. Windows has been using junctions for a long time, especially
when Microsoft decides to change the name of the special folder, like
changing "Documents and Settings", the old name, and "Documents", that
both point to C:\Users. Could be your problematic folder is under a
junction, like Documents.

https://knowledgebase.macrium.com/pages/viewpage.action?pageId=23397420

That gives some information. As I recall, Macrium is supposed to ignore symlinks and junctions when creating backups (to prevent looping). That
is, it still records the reparse points, but it shouldn't follow them.

Make damn sure that Macrium Reflect is *NOT* following reparse points (recording them is okay, but following them during a backup is usually
not okay). Go into Reflect under its Other Tasks menu to select Edit
Defaults. Under the Backup tab, and under the Reparse Points category,
make sure "System - Do not follow" is selected. However, the default
for User Reparse Points is to follow them, but I've seen users screw
them up and generate circular links. See what happens when you set
"User - Do not follow". Those are for the default settings used when
you /create/ a backup job. For old saved job definitions, they may
differ than the current global defaults. Also go into the backup job's definition and set the reparse follow options the same ("Do not follow"
for both system and user defined reparse points).

You could run a test by moving or copying the problematic folder to
elsewhere that is guaranteed not to be under a junction (after first
checking the folder itself has no junctions), like copying the folder to C:\problemfolder, and then having Reflect backup just that folder.

Are the files in the problematic folder in use? If open for write,
another process has to either wait for the file handle to close (get
deleted) or times out. Although I also use Macrium Reflect, configuring
it to run pre- and post-job commands is *very* clumsy. You have to
create a Powershell, VBscript, or batch file and have Macrium run that
as its scheduled task. Once you create the script template, you edit it
to add your own commands before or after the backup job. The problem
that I've run into is that Reflect will have the script run the backup
job by calling Reflect as a service which has admin privileges, but
doesn't load the command shell itself with admin privs in which the
script runs, so commands you enter there that require admin privs won't
run. There might be a way around that, but I gave up on Reflect's
clumsy pre- and post-command workaround feature, plus you have to
maintain the script instead of having an easily configurable command
line to edit in the Reflect GUI when creating or editing a backup job.
However, if you can get Reflect's script feature to work to emulate a
pre- and post-job feature, you might look at running the SysInternals' handle.exe command to see which files might be in-use (have open file
handles) before the backup job starts.

Getting locked out from reading a file can be thwarted by using VSS
(Volume Shadow Service). I'm pretty sure on image backups that Reflect defaults to using VSS. I don't see an option to not use VSS. However,
under Other Tasks menu, Advanced tab, check if Reflect will
"Automatically retry without VSS writers on failure". If there is a
problem with VSS, Reflect will try to backup without VSS.

Also check the VSS service will change into Running status. Go into
Windows services (services.msc), scroll down to "Volume Shadow Copy"
service. It should be set to Manual startup mode, and not Disabled. It
runs when called. It does not stay running during the entire time that
Windows is running. It is only needed when a shadow copy is needed to
get at in-use or system-restricted files, and you are not backing up the
entire time you have Windows loaded. If you go into Event Viewer,
Application logs, and filter on event ID 8224, you'll see informational
events for "The VSS service is shutting down due to idle timeout."

I forget the idle interval, probably 15 minutes, but once started the
VSS service will eventually stop after the last time it got called by a
VSS requestor and after a VSS writer has completed its task. Those
users that whine the VSS has idle-stopped don't understand this service
is not meant to be always running (Automatic mode). It is manually
called by a requestor, used for a while, and then it stops because it's
not being used anymore. Been that way since Microsoft introduced VSS
back in Windows XP to facilitate backing up of in-use and system files.

https://docs.microsoft.com/en-us/windows-server/storage/file-server/volume-shadow-copy-service

Right click on that service and select Start, or select it and click the
Start button. Did it change into Running status (for awhile)?

Some programs install their own VSS writers. As I recall, Paragon
supplied their own optional VSS writer you could select instead of using
the Windows-provided one. Reflect uses the copy-on-write writer already provided by Windows. You can see a list of VSS writers by running in a
command shell:

vssadmin list writers

Sorry, I haven't delved far enough into this to know which system VSS
writer that Reflect will employ. Might be the ASR Writer as noted at https://docs.microsoft.com/en-us/windows/win32/vss/in-box-vss-writers.
Not sure even Reflect cares, as it likely just issues some system API
call to use VSS.

VSS is only usable when NTFS is used as the file system. You didn't
mention WHERE is the problematic folder. If it is a folder on an
internal drive that uses NTFS, VSS can come into play (if the targeted
files are locked). If the folder is on some external storage media,
like a USB HDD or flash drive, could be that uses FAT32 or some other
file system than NTFS, so VSS can't be used there.

If VSS fails when called by Macrium Reflect, the backup job's log should
note the error. See:

https://knowledgebase.macrium.com/display/KNOW72/Troubleshooting+Microsoft+VSS+errors
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan wrote:

On 4/27/2020 3:57 AM, Paul wrote:

I seem to remember at some time in the past, you offered
advice on putting an exception for an AV program,
so it does not scan that particular directory
(something in Thunderbird).

If your CPU cores are railed, I'd be tracing down the
PID of the offender.

One way to do it on a Pro SKU of OS, is

tasklist /svc # should not work on Home

Not even necessary, I can tell you right now which process is
responsible, it's the Macrium Reflect binary. Also the System process
which I assume the Reflect binary also makes heavy use of during this time.

OK, show me a chunk of nfi.exe output, just
for files in the magical folder. Just enough
to capture the essence of what's going on.

nfi.exe is in here (13,529,558 bytes)

https://web.archive.org/web/20070104083656if_/http://download.microsoft.com:80/download/win2000srv/utility/3.0/nt45/en-us/oem3sr2s.zip

Run

nfi.exe C: > c_nfi.txt

This is what a file looks like, followed by a directory.
A directory has a $I30 entry in it.

File 5468

\YOUTUBE_CAP\out_linux_ffmpeg2.avi
$STANDARD_INFORMATION (resident)
$FILE_NAME (resident)
$DATA (nonresident)
logical sectors 2576342736-2577800527 (0x998fded0-0x99a61d4f)

File 5463

\YOUTUBE_CAP
$STANDARD_INFORMATION (resident)
$FILE_NAME (resident)
$INDEX_ROOT $I30 (resident)
$INDEX_ALLOCATION $I30 (nonresident)
logical sectors 2577800616-2577800623 (0x99a61da8-0x99a61daf)
$BITMAP $I30 (resident)

What we're looking for here, is something
like an extended attribute.

You might also use fsutil, and verify the cluster
size (4KB default). Windows 10 stopped tolerating
non-default cluster sizes on C: about three OSes ago,
so it pretty well has to be 4KB now on cluster size.

One reason I want some info about your 800,000 file folder,
is I want to see if there are no logical sectors
(small files, like 1KB files, fit within $MFT and don't
use clusters for the data storage). Or I was io see if
the clusters are fragmented.

One other thing Windows 10 does now, is they added a small
write cache (per handle). The write cache has "ruined" the
notion of fragmentation, in the sense that no fragment
can be 4KB. The buffer is 64KB. If a file fragments today in
Windows 10, the chunk size should be 64KB.

I use the Passmark fragment generator, to create fragmented
files for test. I noticed that if the Passmark fragment
generator is run on modern Windows 10, the fragments
don't seem to be any smaller than 64KB. If I run under an
older OS, you can see on the screen (JKDefrag) that the
fragments are finer. I do these tests on a RAMDisk so
no harm comes to any physical storage devices.

You might ask "I have a 4KB file to store, what happens
with the 64KB buffer in that case". I don't know. Obviously it
cannot break, or we'd have heard about it by now. The buffer must
flush when the handle closes.

I only mention this new feature, in case you examine your
800,000 files and notice there's no fragmentation at all.

Paul
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:

I'll tell you what this folder is. It's actually my Thunderbird News
folder (exactly what I'm using to ask this question here), which exists under the my User folder structure. The problem was discovered when I started doing daily backups of my User folder and discovered that the
User folder was taking forever. After investigating it some, I figured
out that the problem was this particular substructure under News. Once I excluded the News folder, backups finished 6 times faster! So I moved
the backups of the News folder to their own job, and let the rest of the User folder get backed up separately. Before, you ask, I only backup the News folder once a week, but it's still a pain in the ass watching it
take so long even once a week.

Some other background. When this particular backup is happening, it's
not the drives that are showing as busy, it's the CPU cores! 4 out of
the 8 cores on my FX-8300 are fluctuating between 50% to 100% busy,
while the other 4 are not that busy.

Yousuf Khan

As a test, disable your anti-virus software and run your TB data-only
backup job.

As another test, make sure to *exit* Thunderbird (check there are no
instances of TB in Task Manager's Processes tab), and check if the
backup job is just as slow.

Do you leave TB running all the time? Does the backup job run as a
scheduled event at a time after you would've unloaded TB, like you use
TB during the day (say 8AM to 11 PM), unload it when done, and you
schedule the backup job to run early morning (say 4 AM)?

VSS will encounter problems with databases that are not VSS aware.
Microsoft's SQL Server is VSS aware, but others are not. The
recommendation in backup programs, even those using VSS, for database
programs that are not VSS aware is to schedule their shutdown before the backup, schedule the backup while the database program is down, and
restart the database program after the backup finishes. While this can
be done using Task Scheduler using event triggers (provided the database program issues an event on shutdown), it's a pain to figure out the
script-like code you have to use to define for the trigger of the
scheduled event. There are schedulers that are more flexible that can
make their events dependent: task 3 runs only after task 2 ran and
returned good status which runs only after task 1 completed and returned
good status.

https://knowledgebase.macrium.com/display/KNOW72/Backing+up+non-VSS+aware+databases

I sincerely doubt Thunderbird provides its own VSS writer. What does
Tbird use to manage its message store? Isn't it SQLite? SQLite is not
a VSS-aware database program. In fact, it isn't a database program at
all. It's a library from which some program can call its functions (aka methods). It would be up to the calling program to be VSS-aware, and I
doubt Mozilla ever added that to Tbird.

http://sqlite.1065341.n5.nabble.com/SQLite-Support-for-VSS-writer-td85887.html

I remember back when using MS Outlook with POP which stored its message
store in a PST file that backups would often skip that database. While
Outlook was running, its database couldn't be backed up because it
wasn't only in-use but also locked as a database. MS didn't provide a
VSS writer just for Outlook. Some users used batch files that would
kill Outlook, run the backup (to include Outlook's message store), and
reload Outlook after the backup finished. However, Outlook has no way
to gracefully unload it. There is no command-line switch for Outlook to
ask it to unload. You had to kill it, and that's always a bad way to
smash a program with open files since corruption can occur to the files.
Some backup programs worked around the problem by installing an
extension into Outlook that would exit it and start the backup program,
and the backup program would later restart Outlook. I'm sure there were
other workarounds. Since Outlook is a client, not a server, there
really was no need to leave it running 24x7, but a lot of users ran it
that way, so it available upon their return to their computer.

Not all programs that manage a database are VSS-aware. Usually the
easiest solution is to make sure the program using the database is not
running at the time of the backup job. Does Tbird have a command-line
switch that will unload the currently loaded instance(s) of Tbird?
Using taskkill.exe is abrupt and can result in file corruption. If
Tbird can be requested to gracefully shutdown, you could do that in a
script, run the backup job, and reload Tbird (if you can get scripts via Powershell, VBscript, or batch to work in Reflect).

I doubt Tbird generates an event when it exits (i.e., you don't see
anything in Event Viewer). If it does, you can define a scheduled event
in Task Scheduler to run the backup job that triggers on the exit event
of Tbird.
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 4/27/2020 2:03 PM, VanguardLH wrote:

What did you use to check if there were junctions defined within the
folder? For example, you could use Nirsoft's NTFSLinksView tool to scan
for junctions to list them. You can specify the start folder from where
to search, like the folder with the 500K+ files, or search from the root folder of a drive (junctions cannot point to other drives). Alas, if
you pick the problematic folder, a scan will only show any junctions in
that folder, not those that point at that folder. You might want to
scan from the root folder, and then check if that folder is under a
junction. Windows has been using junctions for a long time, especially
when Microsoft decides to change the name of the special folder, like changing "Documents and Settings", the old name, and "Documents", that
both point to C:\Users. Could be your problematic folder is under a junction, like Documents.

I don't have to look for junctions, I know where they are. If there were junctions here, I would have put them in myself, otherwise they aren't
there.

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 4/27/2020 12:04 PM, Frank Slootweg wrote:

If there are 580,000 files in the News folder, then you've probably configured your Thunderbird News account(s) to use one file for each
article instead of one file for each newsgroup.

If so, it's probably best to bite the bullet and convert to one file
per newsgroup. That probably needs an export and (re-)import and
probably will be time-consuming, but at least then you'll solve the
actual problem.

FYI, my setup - not Thunderbird - has nearly a million articles, but
only some 600 files.

Yes, that is exactly the problem, I was getting at. Does Thunderbird
have a new news file format available? My assumption was that
Thunderbird only does 1 file/message? What's the option to convert?

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 4/27/2020 2:29 PM, VanguardLH wrote:

As a test, disable your anti-virus software and run your TB data-only
backup job.

Yes, that's been done years ago too. This folder has been a major
headache for years now. And at one time, I found that the AV software
spending tons of time scanning this folder too, so I put an exclusion in
it for this folder. The AV doesn't ever scan in this folder anymore.

As another test, make sure to*exit* Thunderbird (check there are no instances of TB in Task Manager's Processes tab), and check if the
backup job is just as slow.

Yeah, but it doesn't matter, Thunderbird's email folders don't suffer
from this problem. So even if Thunderbird were running in the
background, and even if it were VSS aware, then this problem would be happening during backups of the email store as well, but it's only
happening in the newsgroup store. The email store is much, much more
active than the newsgroup store, but emails aren't affected, just
newsgroups.

VSS will encounter problems with databases that are not VSS aware. Microsoft's SQL Server is VSS aware, but others are not. The
recommendation in backup programs, even those using VSS, for database programs that are not VSS aware is to schedule their shutdown before the backup, schedule the backup while the database program is down, and
restart the database program after the backup finishes. While this can
be done using Task Scheduler using event triggers (provided the database program issues an event on shutdown), it's a pain to figure out the script-like code you have to use to define for the trigger of the
scheduled event. There are schedulers that are more flexible that can
make their events dependent: task 3 runs only after task 2 ran and
returned good status which runs only after task 1 completed and returned
good status.

Thunderbird never downloads newsgroup messages in the background, like
it does with email, it only downloads them when you explicitly open the newsgroups account. This is also related to what I said above about how
much more busier the Thunderbird email store is compared to the
newsgroup store. Thunderbird may be doing things in the background but
only with email.

It's not related to VSS, I've already given you the most likely cause of
the problem: there are over half million files, and each file is
inefficiently taking up little over half of the NTFS cluster, rather
than spreading a lesser number of files over many clusters. The real
question is how can we make NTFS more efficient at handling all of these little files? NTFS is great at handling big files, but tiny little files
no so much.

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 2020-04-26 18:24, Yousuf Khan wrote:

I have a folder on one of my SSD drives that takes 8 to 10 hours to back
up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.

Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this inefficient for small files like this?

    Yousuf Khan

Hi Yousuf,
When I see things like this, it is usually a failing
drive, especially when the index on teh offending
directory never finishes.
This will show up like a soar thumb if yo run your
drive through gsmartcontrol: check the error logs and
run the self tests http://gsmartcontrol.sourceforge.net/home/index.php/Downloads
Get back to us!
-T
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 2020-04-27 07:30, Ken Blake wrote:

Since the folder is on an SSD, fragmentation shouldn't make any difference.

And you will reduce your wear life doing a defragment
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan wrote:

On 4/27/2020 12:04 PM, Frank Slootweg wrote:

If there are 580,000 files in the News folder, then you've probably
configured your Thunderbird News account(s) to use one file for each
article instead of one file for each newsgroup.

If so, it's probably best to bite the bullet and convert to one file
per newsgroup. That probably needs an export and (re-)import and
probably will be time-consuming, but at least then you'll solve the
actual problem.

FYI, my setup - not Thunderbird - has nearly a million articles, but
only some 600 files.

Yes, that is exactly the problem, I was getting at. Does Thunderbird
have a new news file format available? My assumption was that
Thunderbird only does 1 file/message? What's the option to convert?

Yousuf Khan

These are examples of the setting in the Config Editor.

Note that the GUI "Server Settings" page has the choice grayed
out once the tool is running, implying these can't be switched
on the fly from the GUI.

And changing them here, doesn't mean a "converter" is going to run,
because the tool isn't going to know the "before" and "after"
and figure out what needs to be done, or whether it should
even be doing it.

mail.server.server1.storeContractID = @mozilla.org/msgstore/maildirstore;1

mail.server.server4.storeContractID = @mozilla.org/msgstore/berkeleystore;1

You could try some sort of Import/Export strategy, pulling from
an EML format setup, into an MBOX format setup.

Berkeleystore, as far as I know, is the "file per box" method.
The so-called Mork Storage Format, of which there is
a rudimentary parser available.

Maildirstore, is a file per message method, like an EML at a guess.
You can see this better than I can, as mine are all
going to be Berkeleystore.

Picture of Version 45 or so. Option not available/implemented
before Version 38.

https://i.postimg.cc/Jh7qmhzN/TBird-stores-option.gif

Paul
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

T wrote:

On 2020-04-27 07:30, Ken Blake wrote:

Since the folder is on an SSD, fragmentation shouldn't make any difference. >>

And you will reduce your wear life doing a defragment

One way to do this, is with a Macrium backup and restore,
where you use the forward and back button, go back and
"edit" the size of the destination directory. This
causes the restoration to change restore mode, and it
seems to do a file-by-file write when challenged with
even an insignificant file system size change. You don't
have to "pinch it", and in fact pinching it is not
recommended. Like, make the partition 1MB smaller,
should be enough to trigger file-by-file write mode.

This results in a "mostly defragmented" disk. Due to the
handling of the $MFT and the reserved space for $MFT,
there is some "friction fragmentation" as the reserved
space gets squeezed. I can see a little bit of
fluff that doesn't get fixed. The end result of
changing the partition size on the Macrium restore,
is a mostly defragmented partition.

To save on your SSD while doing experiments like this,
you can test on hard drives, and evaluate using nfi.exe
(mentioned in the thread already).

This approach also places an upper bound on the number
of writes and the amount of flash life you're paying
for the privilege. The Windows 10 Defragmenter is pretty
good, but some of the other defragmenters out there,
they can run all night, and that can't be good for an
SSD.

https://i.postimg.cc/Y0TCt8K5/macrium-as-defragger.gif

I noticed that "side effect" one day after a restore,
where I'd changed the destination partition size. I couldn't
believe what I was seeing. Doing that to the 1.4TB
partition in that picture was mostly a joke, as on
a data partition, you don't really need to do that.
I wanted to see whether it would change the symptoms
of another bug I'm working on (and it didn't help).

Paul
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:

On 4/27/2020 2:29 PM, VanguardLH wrote:

As a test, disable your anti-virus software and run your TB data-only
backup job.

Yes, that's been done years ago too. This folder has been a major
headache for years now. And at one time, I found that the AV software spending tons of time scanning this folder too, so I put an exclusion in
it for this folder. The AV doesn't ever scan in this folder anymore.

I just thought of something else: is that flagged as a special folder? Right-click on the folder, and select Properties. Is there a Customize
tab? If so, select it, and check the setting for "Optimize this folder
for". Set to "General items" (instead of "Pictures").

It's not related to VSS, I've already given you the most likely cause of
the problem: there are over half million files, and each file is inefficiently taking up little over half of the NTFS cluster, rather
than spreading a lesser number of files over many clusters. The real question is how can we make NTFS more efficient at handling all of these little files? NTFS is great at handling big files, but tiny little files
no so much.

Slack space is also a problem with FAT16/32, ext, or other file systems
where AUs (Allocation Units) are clusters or groups of sectors. The
file system will allocate a number of clusters that will encompass the
size of the file, but will be equal to or larger than the file's
content. Slack space is *not* just an NTFS problem.

For NTFS, files under the size for an MFT's file record are stored
inside the MFT since there is already enough space to hold the file
there. Instead of the MFT file record having a pointer to the small
file outside the MFT where there would be a lot of slack space (the
small file is nowhere the size of a cluster), the MFT file record *is*
the file.

An MFT file is 1 KB in size. If the file is smaller than that, the file
is stored in the MFT record. Actually, because the MFT file record has
a fixed 42-byte table at its start and holds file name and system
attributes.

https://hetmanrecovery.com/recovery_news/ntfs-file-system-structure.htm#id4
According to specifications, MFT record size is determined by the
value of a variable in the boot sector. In practical terms, all
current versions of Microsoft Windows are using records sized 1024
bytes. The first 42 bytes store the header. The header contains 12
fields. The other 982 bytes do not have a fixed structure, and are
used to keep attributes.

The MFT is not infinite in size. NTFS has a limit of 4,294,967,295
files per disk (well, per volume). Your 580,000 files is only 0.01% of
NTFS' capacity for file count. Obviously there are lots of files
elsewhere in that volume.

NTFS doesn't have a problem between small and large files regarding
addressing them. It's the level of fragmentation that cause a problem.
Yeah, you think you don't need to defragment and should not defragment
an SSD because, after all, accessing memory at one address is the same
speed as accessing other memory. However, NTFS cannot support an
infinite chain of fragments for a file. Each fragment consumes an
extended file record in the MFT (a record outside the MFT). There are limitations in every file system. Around 1.5 million fragments is the
limit per file under NTFS.

Doesn't Thunderbird have a compaction function? Used it yet? I don't
know if that will eliminate any fragmention of the files used to store
the messages or articles which, as I recall, are stored as seperate
files instead of inside a database, but I haven't used TB in a long
time.

Users don't think they ever need to defragment an SSD. All those extra
writes with no effective change in data content reduces the lifespan of
the SSD (writes are destructive). Sure, when there are few or dozen
fragments then the extra writes to defragment are wasting the SSD. It
takes time to chain from the MFT's record and through every external
extended record (which consumes space in the file system) to build up
the entire file. It's not one lookup in the MFT for the file. It's a
chained lookup for every fragment. IOPS will increase as fragmentation increases, and perhaps why you are seeing high CPU usage when backing up
those files. Most users think of fragmentation as a performance issue
with moving physical media, like hard disks. Fragmentation ON ANY MEDIA
is still an I/O overhead issue and inflates the IOPS to process them
all. Yes, there is a limit in NTFS to the number of fragments that a
file may have, but the more fragments there are the more space is
consumed in the file system to track those fragments and the more CPU
consumed to process the fragments. When an OS sees a file comprised of multiple fragments, there are more multiple I/O operations to process
the whole file. If Windows see 20 pieces at the logical layer, there
are 20 I/O operations to process the whole file as a read or write.

Fragmentation is not just a performance issue at the physical layer. It
is also a performance factor at the logical layer (file system).
Extreme fragmentation requires lots of repeated writes to a file. I
don't know what you've been doing with those files in the problematic
folder. If they are photos, you rarely edit those, just copy them.

Similarly, for a backup job, it has to perform the IOPS'es needed to
read all the files included in the backup. I have under 400,000 files
on my entire OS+app drive (which is a partition spanning the entire
SSD). You have more than that in one folder. From your description,
the backup job is CPU bound with all those IOPS. Do you really have
over 500K files in just one folder? You never considered creating a hierarchical structure of subfolders to hold groups of those files based
on a common criteria for each subfolder? Just because you can dump
hundreds of thousands of files into a single folder doesn't mean that's
a good behavior.

By the say, in Macrium Reflect, did you configure your backup job to
throttle its use of the CPU? That's to prevent a backup job from
sucking up all the CPU while preventing the computer being usable to the
user during the backup. In a Reflect backup job, you can configure its priority. Well, if you set it at max (which is still, I believe, less
than real-time priority), that process sucks up most of the CPU and
leave little for use by other processes making the computer unusable to
you. Even if you schedule the backup to run when you're not at the
computer, other backgrounded processes, like your startup programs, and
even the OS want some CPU slices.

The compression you select for a backup job also dictates how much it
consumes the CPU. You will find very little difference in the size of
the backup file between Medium (recommended) and High compression
levels. The backup job will take a lot longer trying to compress more
the backup file, but the result is little improvement in reduction of
the backup file, especially for non-compressible file formats, like
images, but wastes a lot of CPU time for insignificant gain.

I did not find an option in Reflect to throttle how much bandwidth it
uses on the data bus, like a limit on IOPS. Not for network traffic,
but how busy it keeps the data bus. If it is flooded, and especially
for a high[er] priority process, you have to wait to do any other data
I/O.
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:

On 4/27/2020 12:04 PM, Frank Slootweg wrote:

If there are 580,000 files in the News folder, then you've probably configured your Thunderbird News account(s) to use one file for each article instead of one file for each newsgroup.

If so, it's probably best to bite the bullet and convert to one file
per newsgroup. That probably needs an export and (re-)import and
probably will be time-consuming, but at least then you'll solve the
actual problem.

FYI, my setup - not Thunderbird - has nearly a million articles, but only some 600 files.

Yes, that is exactly the problem, I was getting at. Does Thunderbird
have a new news file format available? My assumption was that
Thunderbird only does 1 file/message? What's the option to convert?

It's not a new News file format, it's a different format.

You set the format in the News account: Tools -> Account settings ->
<your news account> in the left pane -> 'Server Settings' page ->
Message Storage -> Message Store Type:. This field *should* be set to
'File per folder (mbox)'. Yours is probably set to 'File per message (maildir)'.

For your account - i.e. an *existing* account - you probably cannot
change this setting, i.e. you can only set it when you create the
account. Hence my comment about exporting and (re-)importing. If you
cannot change the setting, you will have to export all the articles from
your current account and then re-import all articles into a new account
with 'Message Store Type: File per folder (mbox)'.

The basic Thunderbird program has no export facility and only very
limited import functionality.

For import of e-mail (from Windows Mail), I have used the Thunderbird ImportExportTools [1] Extension, but I have not used it for News and not
for export.

ImportExportTools can export on a per-folder basis, so you could try
to export just one folder/newsgroup and then import it into a new
account to see if it works for News. Exporting is a copy-type operation,
i.e. the source remains untouched, and if you import to a *new* account,
the old account remains untouched. IOW, it's a totally safe operation.

If ImportExportTools can not solve your problem, you'll probably have
to search the Thunderbird support site(s)/forum(s) or/and post there.

[1] <https://addons.thunderbird.net/en-GB/thunderbird/addon/importexporttools/?src=userprofile>
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Frank Slootweg wrote:

Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:

On 4/27/2020 12:04 PM, Frank Slootweg wrote:

If there are 580,000 files in the News folder, then you've probably
configured your Thunderbird News account(s) to use one file for each
article instead of one file for each newsgroup.

If so, it's probably best to bite the bullet and convert to one file
per newsgroup. That probably needs an export and (re-)import and
probably will be time-consuming, but at least then you'll solve the
actual problem.

FYI, my setup - not Thunderbird - has nearly a million articles, but
only some 600 files.

Yes, that is exactly the problem, I was getting at. Does Thunderbird
have a new news file format available? My assumption was that
Thunderbird only does 1 file/message? What's the option to convert?

It's not a new News file format, it's a different format.

You set the format in the News account: Tools -> Account settings ->
<your news account> in the left pane -> 'Server Settings' page ->
Message Storage -> Message Store Type:. This field *should* be set to
'File per folder (mbox)'. Yours is probably set to 'File per message (maildir)'.

For your account - i.e. an *existing* account - you probably cannot
change this setting, i.e. you can only set it when you create the
account. Hence my comment about exporting and (re-)importing. If you
cannot change the setting, you will have to export all the articles from
your current account and then re-import all articles into a new account
with 'Message Store Type: File per folder (mbox)'.

The basic Thunderbird program has no export facility and only very
limited import functionality.

For import of e-mail (from Windows Mail), I have used the Thunderbird ImportExportTools [1] Extension, but I have not used it for News and not
for export.

ImportExportTools can export on a per-folder basis, so you could try
to export just one folder/newsgroup and then import it into a new
account to see if it works for News. Exporting is a copy-type operation,
i.e. the source remains untouched, and if you import to a *new* account,
the old account remains untouched. IOW, it's a totally safe operation.

If ImportExportTools can not solve your problem, you'll probably have
to search the Thunderbird support site(s)/forum(s) or/and post there.

[1] <https://addons.thunderbird.net/en-GB/thunderbird/addon/importexporttools/?src=userprofile>

The one I was looking at the other day, said that it didn't
handle stuff in the News folder specifically.

As for the availability of the MailboxStore option in the
Server settings, the claim is that you must use this
immediately when the installation of Thunderbird is
brand new. In my experiments yesterday, I tried to "clean out"
my profile, and tried not to leave any .msf files, then
set the prefs.js with the maildirstore preference, and
that *still* wasn't enough to make it work. I'm going
to have to nuke the damn thing and start from scratch,
to see if I can get it to work.

One other weirdness from yesterdays experiment, is after
I was finished with my failed experiment, I took the ZIP
file holding my unbroken profile, and started to restore
it to my SSD drive. I was greeted by write rates of arounf
2MB/sec on my SSD. It took forever to restore the fleet
of .msf (file per box) style files. And when I opened
Task Manager, MsMpEng was railed on one core, scanning
everything being written into the profile area. I've done
plenty of other stuff on the computer, where it doesn't
do that with quite the same level of venom. (If I unpack
an .ova on a scratch drive, it does that at several hundred
megabytes per second. As if MsMpEng didn't care.)

Paul
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Paul <nospam@needed.invalid> wrote:
[...]

As for the availability of the MailboxStore option in the
Server settings, the claim is that you must use this
immediately when the installation of Thunderbird is
brand new.

I think that's not correct. The *installation* doesn't have to be
brand new, the *account* in Thunderbird must be new, i.e. just created.

I added a new New account and could set 'Message Store Type:' to
either 'File per folder (mbox)' or 'File per message (maildir)'.

In my experiments yesterday, I tried to "clean out"
my profile, and tried not to leave any .msf files, then
set the prefs.js with the maildirstore preference, and
that *still* wasn't enough to make it work. I'm going
to have to nuke the damn thing and start from scratch,
to see if I can get it to work.

There's no need to fiddle with preferences as there are perfectly good settings in the GUI. I think the fiddling might actually have been counter-productive, because you might have been setting a global default instead of a account-specific setting.

[...]
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 4/27/2020 2:09 PM, Paul wrote:

OK, show me a chunk of nfi.exe output, just
for files in the magical folder. Just enough
to capture the essence of what's going on.

What I'm looking for is filesystem tuning advice, not process tuning
advice. Looking at CPU processes is just a wild goose-chase. Yes, the
CPU is being hammered, but we know which processes are responsible and
why, it's hardly surprising which ones they are (i.e. Macrium Reflect binaries), so it's trivial.

Anyways, since I'm not getting that advice here, as it turned out, I
just received a new SSD (as an RMA of a previous SSD, which has already
been replaced). I decided to try a few tests myself. I set up the new
SSD as the Z drive, and I formatted it into non-default NTFS settings.
This SSD is usually default formatted to 4K blocks, I tested it out by
using 0.5K and 1K blocks instead. I then restored a previous backup of
the filesystem to this drive, and tested out the backup and restore performance. Since this is not the production drive, it's not being
accessed by any other processes like Thunderbird, so it's pristine and
not a busy drive.

I found that with both 0.5K and 1K blocks, the restore operation went
very fast, about half an hour to get fully restored, which is a big improvement (previously used to take 1.5 hours to restore). Also the allocation slack was greatly improved, went from 1.4GB stored and 2.4GB allocated (42% slack), to 1.4GB stored and only 1.5GB allocated (7%
slack). However, then I tried backing up the new drive and it still took
over 8 hours! So writing to the drive is getting very fast, but reading
off of it is still slow. Still the same number of files as before, over half-a-million.

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 4/28/2020 11:13 AM, Frank Slootweg wrote:

Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:

Yes, that is exactly the problem, I was getting at. Does Thunderbird
have a new news file format available? My assumption was that
Thunderbird only does 1 file/message? What's the option to convert?

It's not a new News file format, it's a different format.

New to me. LOL ;-)

You set the format in the News account: Tools -> Account settings ->
<your news account> in the left pane -> 'Server Settings' page ->
Message Storage -> Message Store Type:. This field *should* be set to
'File per folder (mbox)'. Yours is probably set to 'File per message (maildir)'.

Actually it does show the "file per folder (mbox)" but it's completely grey-out, unchangeable. I assume that that's just it's preferred method
of accessing News, but it's being forced to use the old format anyways.

For your account - i.e. an *existing* account - you probably cannot
change this setting, i.e. you can only set it when you create the
account. Hence my comment about exporting and (re-)importing. If you
cannot change the setting, you will have to export all the articles from
your current account and then re-import all articles into a new account
with 'Message Store Type: File per folder (mbox)'.

Actually, I'm ready to completely blow out all of the files in the news folder, and redownload from scratch, just so long as my newsgroups list remains untouched. I obviously have backups of it, so it's not going to
be harmful to me.

If ImportExportTools can not solve your problem, you'll probably have
to search the Thunderbird support site(s)/forum(s) or/and post there.

[1] <https://addons.thunderbird.net/en-GB/thunderbird/addon/importexporttools/?src=userprofile>

Thanks, but it looks like this version only works up to Thunderbird
version 60, and I'm on version 68.x. No problem, this is just News, I'll
just wipe it out and redownload.

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 4/28/2020 3:14 PM, Paul wrote:

The one I was looking at the other day, said that it didn't
handle stuff in the News folder specifically.

As for the availability of the MailboxStore option in the
Server settings, the claim is that you must use this
immediately when the installation of Thunderbird is
brand new. In my experiments yesterday, I tried to "clean out"
my profile, and tried not to leave any .msf files, then
set the prefs.js with the maildirstore preference, and
that *still* wasn't enough to make it work. I'm going
to have to nuke the damn thing and start from scratch,
to see if I can get it to work.

One other weirdness from yesterdays experiment, is after
I was finished with my failed experiment, I took the ZIP
file holding my unbroken profile, and started to restore
it to my SSD drive. I was greeted by write rates of arounf
2MB/sec on my SSD. It took forever to restore the fleet
of .msf (file per box) style files. And when I opened
Task Manager, MsMpEng was railed on one core, scanning
everything being written into the profile area. I've done
plenty of other stuff on the computer, where it doesn't
do that with quite the same level of venom. (If I unpack
an .ova on a scratch drive, it does that at several hundred
megabytes per second. As if MsMpEng didn't care.)

   Paul

Oh, it's a good thing I kept reading the replies, as it looks like you
already tried what I was about to try. So it kept using the same file
format as before, even after nuking it and starting from scratch?
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 4/28/2020 5:08 PM, Frank Slootweg wrote:

I think that's not correct. The*installation* doesn't have to be
brand new, the*account* in Thunderbird must be new, i.e. just created.

I added a new New account and could set 'Message Store Type:' to
either 'File per folder (mbox)' or 'File per message (maildir)'.

So what if I nuke all of the old messages in the News folder, and let it repopulate from scratch?

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 4/27/2020 6:15 PM, T wrote:

Hi Yousuf,

When I see things like this, it is usually a failing
drive, especially when the index on teh offending
directory never finishes.

This will show up like a soar thumb if yo run your
drive through gsmartcontrol: check the error logs and
run the self tests

Brand new drive, less than a month old, hasn't had a chance to get old yet.

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan wrote:

On 4/28/2020 3:14 PM, Paul wrote:

The one I was looking at the other day, said that it didn't
handle stuff in the News folder specifically.

As for the availability of the MailboxStore option in the
Server settings, the claim is that you must use this
immediately when the installation of Thunderbird is
brand new. In my experiments yesterday, I tried to "clean out"
my profile, and tried not to leave any .msf files, then
set the prefs.js with the maildirstore preference, and
that *still* wasn't enough to make it work. I'm going
to have to nuke the damn thing and start from scratch,
to see if I can get it to work.

One other weirdness from yesterdays experiment, is after
I was finished with my failed experiment, I took the ZIP
file holding my unbroken profile, and started to restore
it to my SSD drive. I was greeted by write rates of arounf
2MB/sec on my SSD. It took forever to restore the fleet
of .msf (file per box) style files. And when I opened
Task Manager, MsMpEng was railed on one core, scanning
everything being written into the profile area. I've done
plenty of other stuff on the computer, where it doesn't
do that with quite the same level of venom. (If I unpack
an .ova on a scratch drive, it does that at several hundred
megabytes per second. As if MsMpEng didn't care.)

Paul

Oh, it's a good thing I kept reading the replies, as it looks like you already tried what I was about to try. So it kept using the same file
format as before, even after nuking it and starting from scratch?

I would refrain from working in this direction.

Sure, if you have backed up the various folders for TBird
before trying it (like I did when testing), then great.
Just don't do it, without having something to restore from.

It's pretty weird for a function to be existing in TBird
and presumably to be absorbing test time from release to
release, and then be hobbling the usage of it with
inept controls.

If you pursue this line of reasoning, what will
happen is your headers will be stripped down to
the event horizon of the server (maybe six months
retention on a free server), and if you have
years of headers (where the MID won't fetch anything
if you click), those are the kinds of headers that
will disappear if you start over again. The headers
from ten years ago, aren't on the server, and cannot
be regenerated from a small server - messing around
will significantly damage your header history.

If the damn thing had a conversion function that
converted equally between the two formats, I might
have a different opinion about doing this. It's just
that this is a feature that doesn't appear finished.

Paul

--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 5/1/2020 3:56 AM, Paul wrote:

Yousuf Khan wrote:

Oh, it's a good thing I kept reading the replies, as it looks like you
already tried what I was about to try. So it kept using the same file
format as before, even after nuking it and starting from scratch?

I would refrain from working in this direction.

Sure, if you have backed up the various folders for TBird
before trying it (like I did when testing), then great.
Just don't do it, without having something to restore from.

It's pretty weird for a function to be existing in TBird
and presumably to be absorbing test time from release to
release, and then be hobbling the usage of it with
inept controls.

I just took a chance, and deleted all of the old newsgroup folders, that contained all of the old-style messages. Left all of the rest of the
files in that news server's base folder untouched. Then I started
Thunderbird up again. It re-downloaded the messages, and it only
downloaded from where I last left off. It's now filling the data files
known as *.msf (e.g. alt.comp.os.windows-10.msf) rather than filling the folders! Interestingly, these *.msf files used to exist in this News
folder before, but they were just trivial 1K or 2K files, with nothing substantial inside them. They are now substantial files now, ranging
from 44 KB to 41 MB. So it looks like having those old folders there all
of this time was preventing Thunderbird from using the new style *.msf
files, even though it had long ago created them!

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan wrote:

On 5/1/2020 3:56 AM, Paul wrote:

Yousuf Khan wrote:

Oh, it's a good thing I kept reading the replies, as it looks like
you already tried what I was about to try. So it kept using the same
file format as before, even after nuking it and starting from scratch?

I would refrain from working in this direction.

Sure, if you have backed up the various folders for TBird
before trying it (like I did when testing), then great.
Just don't do it, without having something to restore from.

It's pretty weird for a function to be existing in TBird
and presumably to be absorbing test time from release to
release, and then be hobbling the usage of it with
inept controls.

I just took a chance, and deleted all of the old newsgroup folders, that contained all of the old-style messages. Left all of the rest of the
files in that news server's base folder untouched. Then I started Thunderbird up again. It re-downloaded the messages, and it only
downloaded from where I last left off. It's now filling the data files
known as *.msf (e.g. alt.comp.os.windows-10.msf) rather than filling the folders! Interestingly, these *.msf files used to exist in this News
folder before, but they were just trivial 1K or 2K files, with nothing substantial inside them. They are now substantial files now, ranging
from 44 KB to 41 MB. So it looks like having those old folders there all
of this time was preventing Thunderbird from using the new style *.msf files, even though it had long ago created them!

Yousuf Khan

What I had tested before, was TBird 45 (sufficiently newer than the
TBird 38 that launched maildir). There was no conversion claimed
in TBird 45.

I was just looking at TBird 60.9.1 in a VM here (a setup that's
only used for email testing), and I added a news server to it,
and not only did if offer the button to choose .msf versus
maildir, but when I selected maildir, it claimed to be
"doing a conversion" to the other format. Even though
at that moment, no groups existed.

I added one group, and again it claimed to be doing a
conversion, and now there's a parallel "maildir" folder
which presumes to be a copy of the .msf folder.

If you kept your original setup with the 500000 files,
you might try updating to 60.9.1 or so, and trying
to flip the control using that version. It seemed to
unsubscribe me from the one group I'd selected, but
it seems to have worked. I haven't had time to do much
other testing yet.

Paul
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:
[...]

I just took a chance, and deleted all of the old newsgroup folders, that contained all of the old-style messages. Left all of the rest of the
files in that news server's base folder untouched. Then I started Thunderbird up again. It re-downloaded the messages, and it only
downloaded from where I last left off. It's now filling the data files
known as *.msf (e.g. alt.comp.os.windows-10.msf) rather than filling the folders! Interestingly, these *.msf files used to exist in this News
folder before, but they were just trivial 1K or 2K files, with nothing substantial inside them. They are now substantial files now, ranging
from 44 KB to 41 MB. So it looks like having those old folders there all
of this time was preventing Thunderbird from using the new style *.msf files, even though it had long ago created them!

<boggle!>

If you apparently did not mind to delete all the old articles, then
why did you keep 580,000 old articles in the first place!?

You can set global and per group retention policies, so if you do not
need so much articles, just set those to appropriate values.
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 5/1/2020 10:55 AM, Frank Slootweg wrote:

<boggle!>

If you apparently did not mind to delete all the old articles, then
why did you keep 580,000 old articles in the first place!?

Simple, because I had no idea what the purpose of any of these files in
this folder were for in any detail, what was important, and where
exactly data resided, so I just backed up everything. That way I
wouldn't have to recreate everything from scratch, and go through hours
of debugging. I've had situations were just 1 important file goes
missing which screws up the entire configuration, and trying to find
that one missing file among half million is a needle in a haystack.

So now after the deletion, I'm down from half million to only about 600
files. And I did a test backup, and the backup went from over 8 hours,
down to only 2.5 minutes! My feeling is that perhaps a lot of those half-million files were just left over from decades of junk that
Thunderbird did not clear, even though it said it was clearing them.

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 5/1/2020 5:11 AM, Paul wrote:

If you kept your original setup with the 500000 files,
you might try updating to 60.9.1 or so, and trying
to flip the control using that version. It seemed to
unsubscribe me from the one group I'd selected, but
it seems to have worked. I haven't had time to do much
other testing yet.

Well, I have been completely uptodate on the Thunderbird releases for a
while now. I was running 68.7, even before this.

I think what's happening here is that Thunderbird wasn't expecting there
to be such long-term'ers like me continuously using their product. I'd
been using Thunderbird since version 0.something, and what was around
back then, is not what is around now, and so they never expected that
I'd be around since back then, and they had no plans for how to migrate old-timers like me. So they just kept using the old format files in my
setup, even though the new format already existed, but they just ignored it.

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:

On 5/1/2020 10:55 AM, Frank Slootweg wrote:

<boggle!>

If you apparently did not mind to delete all the old articles, then
why did you keep 580,000 old articles in the first place!?

Simple, because I had no idea what the purpose of any of these files in
this folder were for in any detail, what was important, and where
exactly data resided, so I just backed up everything. That way I
wouldn't have to recreate everything from scratch, and go through hours
of debugging. I've had situations were just 1 important file goes
missing which screws up the entire configuration, and trying to find
that one missing file among half million is a needle in a haystack.

I can - sort of - understand that, but because these 580,000 were
giving you so much hardship, I would have expected you to look at a
few of them, see that they were just News articles and take it from
there, i.e. set/lower the News retention settings in Thunderbird.

So now after the deletion, I'm down from half million to only about 600 files. And I did a test backup, and the backup went from over 8 hours,
down to only 2.5 minutes! My feeling is that perhaps a lot of those half-million files were just left over from decades of junk that
Thunderbird did not clear, even though it said it was clearing them.

What you saw about "clearing" (the actual term is 'Compact'(ing)) is
for e-mail, not for News. This was already mentioned in this thread,
IIRC by VanguardLH. E-mail folders need to be compacted, because you
might delete some messages from a folder, so the .msf file needs to be compacted to recover the space occupied by the deleted messages. News
articles can be deleted as well (in Thunderbird), but most people won't, because there's no point, because you can only delete your *copy*, not
the copies on the rest of The Net.

Anyway, you should probably set the (News) retention settings,
otherwise the storage will grow again without bounds, not not in number
of files, but in number of MBs/GBs.
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 5/1/2020 1:52 PM, Frank Slootweg wrote:

I can - sort of - understand that, but because these 580,000 were
giving you so much hardship, I would have expected you to look at a
few of them, see that they were just News articles and take it from
there, i.e. set/lower the News retention settings in Thunderbird.

No, I knew those were the message files, considering that there were so
many of them, what else could they have been? But often there are other
files interspersed among them, that can often go overlooked because it's overwhelmed by the mass of all of the main files. Just let the backup
software handle backing all of it up.

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 2020-05-01 00:24, Yousuf Khan wrote:

On 4/27/2020 6:15 PM, T wrote:

Hi Yousuf,

When I see things like this, it is usually a failing
drive, especially when the index on teh offending
directory never finishes.

This will show up like a soar thumb if yo run your
drive through gsmartcontrol: check the error logs and
run the self tests

Brand new drive, less than a month old, hasn't had a chance to get old yet.

    Yousuf Khan

means nothing. test it
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:

On 5/1/2020 1:52 PM, Frank Slootweg wrote:

I can - sort of - understand that, but because these 580,000 were
giving you so much hardship, I would have expected you to look at a
few of them, see that they were just News articles and take it from
there, i.e. set/lower the News retention settings in Thunderbird.

No, I knew those were the message files, considering that there were so
many of them, what else could they have been? But often there are other files interspersed among them, that can often go overlooked because it's overwhelmed by the mass of all of the main files. Just let the backup software handle backing all of it up.

Yes, but "Just let the backup software handle backing all of it up."
was the *problem* which made you start this thread! You can't have it
both ways, it either was a problem/annoyance/<whatever>, or it wasn't!

As to "But often there are other files interspersed among them, that
can often go overlooked because it's overwhelmed by the mass of all of
the main files.", I didn't say to bluntly clobber all the *files*/
*folders*, but to set/lower the News retention settings in Thunderbird.
I.e. let *Thunderbird* do it *safely*, instead of you doing it
(possibly) unsafely.

You *do* know how to set/lower the News retention settings in
Thunderbird, don't you!? (You snipped my other comments about that, so I
don't know if you've set/lowered them now.)
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 5/2/2020 9:45 AM, Frank Slootweg wrote:

Yes, but "Just let the backup software handle backing all of it up."
was the*problem* which made you start this thread! You can't have it
both ways, it either was a problem/annoyance/<whatever>, or it wasn't!

Have what both ways?? I wanted the data backed up, and it was doing
that, but slowly. That's what the problem was that I wanted fixed.

As to "But often there are other files interspersed among them, that
can often go overlooked because it's overwhelmed by the mass of all of
the main files.", I didn't say to bluntly clobber all the*files*/
*folders*, but to set/lower the News retention settings in Thunderbird.
I.e. let*Thunderbird* do it*safely*, instead of you doing it
(possibly) unsafely.

You*do* know how to set/lower the News retention settings in
Thunderbird, don't you!? (You snipped my other comments about that, so I don't know if you've set/lowered them now.)

Hard to say what happened in the deep past. At some point perhaps
Thunderbird was taking so long to delete old messages, so it may have
been locking up or fail-exiting constantly, especially at a time when
this may have been running on slow HDD's rather than SSD's. So a bandaid solution may have come up to prevent it from doing any further
deletions, which got implemented, it stabilized the system, and then
forgotten about. The problem with having run with something for so many decades, previous problems are not even in memory anymore. I don't even
know if this is what happened, that's just my guess at this point.

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.112

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 4/26/2020 9:24 PM, Yousuf Khan wrote:

I have a folder on one of my SSD drives that takes 8 to 10 hours to back
up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.

Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this inefficient for small files like this?

    Yousuf Khan

Okay, so after fixing the problem with my News folder, I kept
researching what these millions of little files were, that were clogging
up my News folder. The files had an extension of WDSEML. Later I found
out that these same files are also there in Email folders, hundreds of thousands of them too.

Initially, I thought that these must be the bodies of the messages that Thunderbird uses to store emails and newsgroup messages. But after a bit
of research, I found out that Thunderbird itself has no use for these
files. Thunderbird does generate them, but it doesn't use them itself.
Instead it is generated only for the benefit of Windows' Search and
Indexing application. Windows Search uses it to be able to let you
search messages through the Windows Search box. So once Thunderbird
generates these files for Windows Search, it no longer has any use for
them anymore, as it stores its own internal data in a different set of
files. In fact, these WDSEML files are saved copies of individual
messages out of Thunderbird's own database. So Thunderbird maintains it
own database, but it never cleans up these copies ever in its life.
WDSEML means "Windows Desktop Search Email", in fact. I also think this
is only a specific problem with Thunderbird under Windows, it probably
isn't an issue in Thunderbird under other OS'es like Linux.

You can easily delete all of these messages, but of course Thunderbird
will regenerate them again as they come in. So what you have to do is
tell Thunderbird not to generate these files for Windows anymore. You go
into Thunderbird's options menu and turn it off (Tools → Options, then select Advanced → General → System Integration → Allow Windows search to search messages).

https://fileinfo.com/extension/wdseml

You can also delete them more easily by searching for and deleting just
the folders in which they reside, rather than the individual files.
These folders have an extension called *.MOZMSGS.

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.113

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan wrote:

On 4/26/2020 9:24 PM, Yousuf Khan wrote:

I have a folder on one of my SSD drives that takes 8 to 10 hours to
back up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.

Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this
inefficient for small files like this?

Yousuf Khan

Okay, so after fixing the problem with my News folder, I kept
researching what these millions of little files were, that were clogging
up my News folder. The files had an extension of WDSEML. Later I found
out that these same files are also there in Email folders, hundreds of thousands of them too.

Initially, I thought that these must be the bodies of the messages that Thunderbird uses to store emails and newsgroup messages. But after a bit
of research, I found out that Thunderbird itself has no use for these
files. Thunderbird does generate them, but it doesn't use them itself. Instead it is generated only for the benefit of Windows' Search and
Indexing application. Windows Search uses it to be able to let you
search messages through the Windows Search box. So once Thunderbird generates these files for Windows Search, it no longer has any use for
them anymore, as it stores its own internal data in a different set of files. In fact, these WDSEML files are saved copies of individual
messages out of Thunderbird's own database. So Thunderbird maintains it
own database, but it never cleans up these copies ever in its life.
WDSEML means "Windows Desktop Search Email", in fact. I also think this
is only a specific problem with Thunderbird under Windows, it probably
isn't an issue in Thunderbird under other OS'es like Linux.

You can easily delete all of these messages, but of course Thunderbird
will regenerate them again as they come in. So what you have to do is
tell Thunderbird not to generate these files for Windows anymore. You go into Thunderbird's options menu and turn it off (Tools → Options, then select Advanced → General → System Integration → Allow Windows search to
search messages).

https://fileinfo.com/extension/wdseml

You can also delete them more easily by searching for and deleting just
the folders in which they reside, rather than the individual files.
These folders have an extension called *.MOZMSGS.

Yousuf Khan

In the business that would be called a "lazy implementation".

All they would have to do, is write a "search provider" and Windows
could use that to pump the files in an OLE fashion. It could have
been done by making no temporary files at all (flow from MORK file
or MBOX or whatever, right into the Windows.edb, in terms of writes).

But that would also put too much Windows-ecosystem code into
the tool, which is a no-no in cross platform tool design. You
have to keep your "philosophical purity" at all costs. Which means
using OpenGL for graphics (cross platform), instead of DirectX and X11
as separate platform interfaces.

I guess there's some benefit to federated search that includes
your email, but to my way of thinking this would only clutter up
a search result later.

Then you'd find yourself typing this in the File Explorer search box:

file:mytaxes.xlsx

instead of

mytaxes

because in the latter one, 500K of your emails are
going to get searched too. Using the file: keyword
would help staunch the mess inside the federated
database. The second search, the results would likely
scroll off the screen, obscuring the thing you really
wanted.

You might also discover the Windows.edb file is bloated
beyond recognition, because of that file set. It might
range around 1GB for a vanilla install, but after that
Thunderbird thing got indexed, it would likely double
at the very least.

You can rebuild the Windows.edb index file, using
the Indexing Options control panel in Windows 10.
I would give that a whirl after the TB folder has
had all the cruft removed. It'll take about three
hours to index the regular C: files (but this assumes
you've customized the searched folders to include
most of C: , versus the very shallow folder set used
by default).

Even finding Windows.edb is hard :-) The File Explorer
search won't allow you to find it. You'll need Agent
Ransack or Everything.exe to find that file, just so
you can see the current size, and decide whether it
needs a rebuild or not.

Aren't computers wonderful ? Such labor saving. "It
slices, it dices, it makes Julienne Fries." I don't
think I've ever made Julienne Fries, but I bet
Windows 10 has done all the pre-work for that,
over and over and over again...

Paul
--- Synchronet 3.18a-Linux NewsLink 1.113

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:

Okay, so after fixing the problem with my News folder, I kept
researching what these millions of little files were, that were
clogging up my News folder. The files had an extension of WDSEML.
Later I found out that these same files are also there in Email
folders, hundreds of thousands of them too.

Initially, I thought that these must be the bodies of the messages
that Thunderbird uses to store emails and newsgroup messages. But
after a bit of research, I found out that Thunderbird itself has no
use for these files. Thunderbird does generate them, but it doesn't
use them itself. Instead it is generated only for the benefit of
Windows' Search and Indexing application. Windows Search uses it to
be able to let you search messages through the Windows Search box. So
once Thunderbird generates these files for Windows Search, it no
longer has any use for them anymore, as it stores its own internal
data in a different set of files. In fact, these WDSEML files are
saved copies of individual messages out of Thunderbird's own
database. So Thunderbird maintains it own database, but it never
cleans up these copies ever in its life. WDSEML means "Windows
Desktop Search Email", in fact. I also think this is only a specific
problem with Thunderbird under Windows, it probably isn't an issue in Thunderbird under other OS'es like Linux.

You can easily delete all of these messages, but of course
Thunderbird will regenerate them again as they come in. So what you
have to do is tell Thunderbird not to generate these files for
Windows anymore. You go into Thunderbird's options menu and turn it
off (Tools → Options, then select Advanced → General → System Integration → Allow Windows search to search messages).

https://fileinfo.com/extension/wdseml

You can also delete them more easily by searching for and deleting
just the folders in which they reside, rather than the individual
files. These folders have an extension called *.MOZMSGS.

Interesting find. I don't remember looking at this option when I
previously trialed Thunderbird. Is this option enabled by default? If
so, a very bad choice my Mozilla.

If I had not known about this option (and I was still using
Thunderbird), and after finding the superfluous and unwanted wdseml
files (since I do *not* want Windows search looking into my e-mails to
confuse those hits with those of files where I want to find by name or content), I probably would've added them to the Include option in
CCleaner which I sometimes run manually but is also a daily scheduled
event in Task Scheduler to run before the daily backup. I have other
programs that leave shit behind that I want purged, so I go into
CCleaner's options, Include section, and define a template of what to
include in CCleaner's cleanup. Some programs, for example, will save
files for a 'resume' function, like a downloader, to continue the
operation when I next load the program. Nope, if I killed/exited the
program then I do not want it wasting time when I next load the program.
I don't even let my web browsers resume a prior session, and configure
them to purge all local data upon their exit.

Back when I used MS Outlook, it was configured by default to allow
Windows Search to look inside my e-mails. No thanks. I disabled that.
If I want to search my e-mails, I'll do that search from inside the
e-mail program. I don't want e-mails mixed in with other file results
in a global search. As I recall, Outlook's search would bitch with an
info insert at the top of the search results that I had Windows Search
disabled for Outlook, but that's exactly how I wanted it to work.

In addition, I used auto-archiving in Outlook not only to move old
e-mails into an archive store, but also to expire and delete very old
e-mails. When they get over 5 years old, I don't need them anymore. I
had auto-archive move messages older than 1 year into the archive, and
had auto-archive delete messages older than 5 years in the archive.
Actually I chained archives together for different expirations: archive messages older than 1 year into archive1year, archive messages older
than 2 years from archive1year into archive2year, and so on. Eventually
I decided I didn't need that level of granularity for storing old
messages, and just went with a single archive for anything older than a
year but purged anything older than 5 years from the archive. I
certainly would not want those old and deleted e-mails still lingering
in a search database or, in your case with Thunderbird, lingering around
in wdseml files.

To me, having Windows Search dig around inside everything is for those
boobs that are slobs. They haven't a clue how to organize their data,
or are too lazy to do it. They pile thousands of e-mails into the Inbox
folder instead of organize the old e-mails into separate pending or
archive folders, and God forbid they delete old e-mails. They'll pile thousands of image files into a single folder instead of use folders to organize them. Foldering is an organizational feature that some users
just seem incapable or unwilling to use. As disorganized is their data
is probably the same for how disorderly is their home.

I'm a bit surprized Mozilla, in developing a cross-platform product,
whould give a gnat's fart about kowtowing to Microsoft's search feature
in Windows. Hell, Mozilla doesn't even use the global certificate store
in Windows (use certmgr.msc to see) within Firefox, and instead uses a
private cert store inside of Firefox (and why some programs have to do a
double cert install: once into the Windows global cert store and again
into Firefox's private cert store). If users are going to search their
old e-mails, why would they not do that from inside of Thunderbird?
They're searching on e-mails, not on some pic they stored from their
camera or a copy of their tax form. Overreaching got even worse in
Windows 10 with Cortana (which I disabled). All this forensic-like
searching to cater to data slobs.
--- Synchronet 3.18a-Linux NewsLink 1.113

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 5/11/2020 4:23 PM, VanguardLH wrote:

Interesting find. I don't remember looking at this option when I
previously trialed Thunderbird. Is this option enabled by default? If
so, a very bad choice my Mozilla.

I have no idea, when it came in, nor when it became an option. That's
the problem with using a program for so many years and decades, you stop looking at its configuration, and accept it doing things by default.

One thing I did find out about this option is that it can be set in two separate places within the Thunderbird options menu, and that they are
not synchronized with each other, for some reason! In one subsection of
the options, it was shown as not selected, but in a different submenu it
shows up again, and it was selected! So I just unselected in both
places, I don't have time to figure out what the differences are, or why
it's in two places. I just hope it's not in 3 places! I assume that
there was some kind of a redesign of the options interface, and so
somebody decided to move the location of this option, but may have
forgotten to remove it from the original place. This may be one of the problems you run into due to this being an amateur collaborative design effort, and there's not a unified design goal. I won't say this is only
a problem with amateur projects, Microsoft itself does plenty of these
things, you have to relearn Office or Windows everytime you upgrade it.

If I had not known about this option (and I was still using
Thunderbird), and after finding the superfluous and unwanted wdseml
files (since I do*not* want Windows search looking into my e-mails to confuse those hits with those of files where I want to find by name or content), I probably would've added them to the Include option in
CCleaner which I sometimes run manually but is also a daily scheduled
event in Task Scheduler to run before the daily backup. I have other programs that leave shit behind that I want purged, so I go into
CCleaner's options, Include section, and define a template of what to
include in CCleaner's cleanup. Some programs, for example, will save
files for a 'resume' function, like a downloader, to continue the
operation when I next load the program. Nope, if I killed/exited the
program then I do not want it wasting time when I next load the program.
I don't even let my web browsers resume a prior session, and configure
them to purge all local data upon their exit.

That's interesting, I did not realize that CCleaner can be custom
configured to get rid of whatever files you like?

But actually regarding getting rid of these files themselves. For years
I was fooled into thinking that they were actually important files that Thunderbird uses. You take one of these files and open it in a text
editor, and you see right away that it looks like an email or a
newsgroup message, so you easily think that this is how those messages
are stored in Thunderbird. So I didn't dare to delete them.

To me, having Windows Search dig around inside everything is for those
boobs that are slobs. They haven't a clue how to organize their data,
or are too lazy to do it. They pile thousands of e-mails into the Inbox folder instead of organize the old e-mails into separate pending or
archive folders, and God forbid they delete old e-mails. They'll pile thousands of image files into a single folder instead of use folders to organize them. Foldering is an organizational feature that some users
just seem incapable or unwilling to use. As disorganized is their data
is probably the same for how disorderly is their home.

Microsoft does a ton of intrusive or esoteric things that it thinks are
stuff users want, but nobody does, and nobody ends up using them in the
end. Then Microsoft removes them, much to the consternation of the
couple of tenths of a percent of people who did use them and found them useful. For example, goodbye Homegroups in Windows networking, or the
ignoring of folder Libraries nowadays. Both of these were features that
came in with Windows Vista or 7, and are going away already. I liked
both of those features, and it annoys me that they are going away.

I'm a bit surprized Mozilla, in developing a cross-platform product,
whould give a gnat's fart about kowtowing to Microsoft's search feature
in Windows. Hell, Mozilla doesn't even use the global certificate store
in Windows (use certmgr.msc to see) within Firefox, and instead uses a private cert store inside of Firefox (and why some programs have to do a double cert install: once into the Windows global cert store and again
into Firefox's private cert store). If users are going to search their
old e-mails, why would they not do that from inside of Thunderbird?
They're searching on e-mails, not on some pic they stored from their
camera or a copy of their tax form. Overreaching got even worse in
Windows 10 with Cortana (which I disabled). All this forensic-like
searching to cater to data slobs.

Well, as Paul mentioned, Mozilla actually didn't really do much to
integrate it into Thunderbird. All they did was create tons of little
text files which Windows Search can then look through, instead of
directly integrating the Windows search API so that Windows can look
directly into the Thunderbird database itself. Lazy programming. Maybe
it was their attempt at competing against Microsoft Outlook, which did integrate the Windows search API, so that it can say that Windows search
also works across Thunderbird?

In the meantime, I was sitting here completely unaware of all of these
useless features that were clogging up my backups. Just as I was typing
this, one of my backups started in the background and it is already
finished, in about a half-hour. Previously this backup used to take
between 1.5 to 2.5 hours! In fact, I can probably reintegrate the News
backups into my main scheduled backup again, which I had to separate out
years ago, due to how long it was taking. If I had done nothing, I would
have eventually had to remove my Emails folder from the main backup too, because that folder was starting to buildup with this crud too.

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.113

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 5/11/2020 1:54 PM, Paul wrote:

In the business that would be called a "lazy implementation".

Absolutely, when I used to program, this is the sort of hacky
programming I'd see other programmers implementing, and it would be my
job to clean this up. I did not know that Mozilla would be doing exactly
the same sort of thing as you'd see a local office programmer doing.

All they would have to do, is write a "search provider" and Windows
could use that to pump the files in an OLE fashion. It could have
been done by making no temporary files at all (flow from MORK file
or MBOX or whatever, right into the Windows.edb, in terms of writes).

Yeah, I don't blame them for not getting too integrated into the Windows ecosystem. If it's something that all operating systems provide in some
form or another, then it can be generalized through a standard
C-library, and they wouldn't have to do special implementations for each
OS. Are there similar functions available in other competing OSes, like
the Microsoft OLE?

I guess there's some benefit to federated search that includes
your email, but to my way of thinking this would only clutter up
a search result later.

I'm betting that they created this hacky implementation, just to say
that they can integrate into Windows Search, just like their competitor Microsoft Outlook does. The WDSEML file is nothing more than their
standard EML file, which is their way of exporting individual messages
to a text format that you can transfer around easily.

You might also discover the Windows.edb file is bloated
beyond recognition, because of that file set. It might
range around 1GB for a vanilla install, but after that
Thunderbird thing got indexed, it would likely double
at the very least.

Yeah, I often see Windows.edb getting hammered when looking through the Windows Resource Monitor app.

You can rebuild the Windows.edb index file, using
the Indexing Options control panel in Windows 10.
I would give that a whirl after the TB folder has
had all the cruft removed. It'll take about three
hours to index the regular C: files (but this assumes
you've customized the searched folders to include
most of C: , versus the very shallow folder set used
by default).

No, I think I'll look into optimizing Windows.edb sometime in the
distant future, at this point in time, I'm done with optimizing my
filesystem. Until another problem arises. :-)

Even finding Windows.edb is hard :-) The File Explorer
search won't allow you to find it. You'll need Agent
Ransack or Everything.exe to find that file, just so
you can see the current size, and decide whether it
needs a rebuild or not.

Well, even Agent Ransack didn't see it, but I assume I'll need to run it
as admin.

The Windows Search is so braindead. You can tell it to search for
something like "data", but it won't find words like "database" even
though data is in the name. You'd have to directly search for "database"
to find anything in the Windows search, no wonder people keep using
Agent Ransack.

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.113

From Newsgroup: comp.sys.ibm.pc.hardware.storage

VanguardLH <V@nguard.lh> wrote:

Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:

[...]

[About .wdseml (Windows Desktop Search Email) files/messages:]

You can easily delete all of these messages, but of course
Thunderbird will regenerate them again as they come in. So what you
have to do is tell Thunderbird not to generate these files for
Windows anymore. You go into Thunderbird's options menu and turn it
off (Tools ? Options, then select Advanced ? General ? System
Integration ? Allow Windows search to search messages).

https://fileinfo.com/extension/wdseml

You can also delete them more easily by searching for and deleting
just the folders in which they reside, rather than the individual
files. These folders have an extension called *.MOZMSGS.

Interesting find. I don't remember looking at this option when I
previously trialed Thunderbird. Is this option enabled by default? If
so, a very bad choice my Mozilla.

No, the option is off by default.
--- Synchronet 3.18a-Linux NewsLink 1.113

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Frank Slootweg <this@ddress.is.invalid> wrote:

VanguardLH <V@nguard.lh> wrote:

Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:

[...]

[About .wdseml (Windows Desktop Search Email) files/messages:]

You can easily delete all of these messages, but of course
Thunderbird will regenerate them again as they come in. So what you
have to do is tell Thunderbird not to generate these files for
Windows anymore. You go into Thunderbird's options menu and turn it
off (Tools ? Options, then select Advanced ? General ? System
Integration ? Allow Windows search to search messages).

https://fileinfo.com/extension/wdseml

You can also delete them more easily by searching for and deleting
just the folders in which they reside, rather than the individual
files. These folders have an extension called *.MOZMSGS.

Interesting find. I don't remember looking at this option when I
previously trialed Thunderbird. Is this option enabled by default? If
so, a very bad choice my Mozilla.

No, the option is off by default.

https://bugzilla.mozilla.org/show_bug.cgi?id=552769

I see someone requested all those .wdseml files (under the .mozmsgs
folders) get deleted if the "Allow Windows Search" option gets disabled.
Opened on 10 YEARS AGO! Status is still New. Geezus.

https://bugzilla.mozilla.org/show_bug.cgi?id=1249056

Tbird will hang at times when moving a folder with IMAP. But then
Windows Search seems to have problems when IMAP items are moved, as
noted at:

https://bugzilla.mozilla.org/show_bug.cgi?id=567212

Then at:

https://bugzilla.mozilla.org/show_bug.cgi?id=553048

users try to disable the option but it immediately reenables itself.
Rude! Provide the option but do not honor the user's choice. Yousuf
needs to check if the option: (1) remains disabled across multiple
restarts of Thunderbird; and, (2) if the option remains enabled if the
.wdseml files that he deleted are not replace with newly generated
.wdseml files.

The more I have dig into Thunderbird and its bugs whether reported or
not, the more I get the feeling that the "developers" are CSCI
undergraduates, and over the years the turn over of volunteers resulted
in no old farts left that are intimate with the entire product. When
Mozilla declared it was considering dumping Thunderbird onto other
open-source organizations (like how OpenOffice got dumped at the Apache Software Foundation) was when I decided to terminate my trial of
Thunderbird. Well, that and my exasperation with Thunderbird that
pushed me to also dump it after a 6-month trial.

https://www.zdnet.com/article/mozilla-we-will-keep-thunderbird-after-all-so-long-as-its-not-a-burden-to-firefox/

"pull the plug with six months' notice if the Thunderbird project does
not make "meaningful progress in short order" in creating technical
infrastructure that's independent of Mozilla Corporation's."

That article is dated back in 2017. So, what magical evolution in
development resources has occurred for Thunderbird in the meantime?

"Mozilla stopped throwing resources at the project in 2012"

Somewhat explains why a vast number of big tickets have never been
addressed, but there are tickets still listed as New dating back to
2004. From https://en.wikipedia.org/wiki/Mozilla_Thunderbird#History,
lots of wavering on what to do with this lead balloon.
--- Synchronet 3.18a-Linux NewsLink 1.113

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 12/05/2020 6:25 AM, Frank Slootweg wrote:

No, the option is off by default.

It's off in one section of the options, but it's on in another section.

--
Sent from Giganews on Thunderbird on my Toshiba laptop
--- Synchronet 3.18a-Linux NewsLink 1.113

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 12/05/2020 7:41 AM, VanguardLH wrote:

https://bugzilla.mozilla.org/show_bug.cgi?id=552769

I see someone requested all those .wdseml files (under the .mozmsgs
folders) get deleted if the "Allow Windows Search" option gets disabled. Opened on 10 YEARS AGO! Status is still New. Geezus.

Good find! And I see that there were discussions just 18 days ago on
this bug, and they were basically saying that considering how old this
request was made, that should he assume that this will never get
implemented?

Then at:

https://bugzilla.mozilla.org/show_bug.cgi?id=553048

users try to disable the option but it immediately reenables itself.
Rude! Provide the option but do not honor the user's choice. Yousuf
needs to check if the option: (1) remains disabled across multiple
restarts of Thunderbird; and, (2) if the option remains enabled if the .wdseml files that he deleted are not replace with newly generated
.wdseml files.

Yes, this option appears in two separate sections, and I think as long
as it's off in both places, then it stays off. I just implemented this
on my laptop, after doing the same on my desktop. It's stayed off in
both places in both computers. It appears under Options -> Advanced ->
General => System Integration -> Allow Windows Search to search
messages. And secondly, it also appears right above that option under
"Always check to see if Thunderbird is the default mail client on
startup", and there is a "Check Now" button; when you press the Check
Now, then a dialog box appears with this option enabled by default,
which you can uncheck.

https://www.zdnet.com/article/mozilla-we-will-keep-thunderbird-after-all-so-long-as-its-not-a-burden-to-firefox/

"pull the plug with six months' notice if the Thunderbird project does
not make "meaningful progress in short order" in creating technical
infrastructure that's independent of Mozilla Corporation's."

That article is dated back in 2017. So, what magical evolution in development resources has occurred for Thunderbird in the meantime?

"Mozilla stopped throwing resources at the project in 2012"

Somewhat explains why a vast number of big tickets have never been
addressed, but there are tickets still listed as New dating back to
2004. From https://en.wikipedia.org/wiki/Mozilla_Thunderbird#History,
lots of wavering on what to do with this lead balloon.

I think both email and newsgroups are now considered "old school"
Internet, few of the kids want to use either one of these anymore. When
they do use email, they usually use it through a web interface, rather
than through a locally stored interface.

--
Sent from Giganews on Thunderbird on my Toshiba laptop
--- Synchronet 3.18a-Linux NewsLink 1.113

From Newsgroup: comp.sys.ibm.pc.hardware.storage

VanguardLH wrote:

The more I have dig into Thunderbird and its bugs whether reported or
not, the more I get the feeling that the "developers" are CSCI undergraduates, and over the years the turn over of volunteers resulted
in no old farts left that are intimate with the entire product.

I expect the team is smaller than the one in the photo here.

https://blog.thunderbird.net/2014/11/thunderbird-reorganizes-at-2014-toronto-summit/

Paul
--- Synchronet 3.18a-Linux NewsLink 1.113

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:

On 12/05/2020 6:25 AM, Frank Slootweg wrote:

No, the option is off by default.

It's off in one section of the options, but it's on in another section.

Nope, it's off there [1] as well.

I've been using Thunderbird for 10 years and there are no .wdseml
files, no .mozmsgs folders and the 'Allow Windows Search to search
messages' option is unticked in both [2] places.

[1] Where 'there' is as you mentioned:

Tools -> Options -> 'Advanced' major tab -> 'General' sub-tab -> System Integration -> Always check to see if Thunderbird is the default mail
client on startup -> Check Now.... -> In the 'System Integration' popup,
the 'Allow Windows Search to search messages' option is *not* ticked.

[2] It's of course [3] one option, which you can set or unset in these
two places.

[3] OTOH, considering this 'Mozilla' crap, 'of course' is a rather
tricky notion! :-(
--- Synchronet 3.18a-Linux NewsLink 1.113

From Newsgroup: comp.sys.ibm.pc.hardware.storage

VanguardLH <V@nguard.lh> wrote:
[...]

The more I have dig into Thunderbird and its bugs whether reported or
not, the more I get the feeling that the "developers" are CSCI undergraduates, and over the years the turn over of volunteers resulted
in no old farts left that are intimate with the entire product.

Thunderbird is an unbelievable mess and they keep on messing with it ('enhancements' :-(), instead of - as you pointed out. fixing bugs.

One of my many pet peeves: Thunderbird has 35+ - yes *thirty five
plus* - update-related settings (app.update.*) and *still* you can't
tell it to only notify you of updates *once* and then shut up (and not
install said updates).

I've set my old (60.9.0) Thunderbird to 'Never check for updates' and
managed to get rid of an already downloaded (but not installed) update,
so that it does no longer bug me about that.

But on SWMBO's system updates sneaked in and the 'Never check for
updates' option is no longer there! :-( So it's "We'll keep bugging you
till you give in or click the 'wrong' button and rinse-and repeat ever
after!". :-(

I've renamed Thunderbird's update.exe on her system, and managed to
get rid of the 'pop-down' (I think Mozilla calls it a 'doorhanger') she
gets at startup, nagging her to download a new version of Thunderbird,
because hers (68.5.0) can not be updated.

On this is just *one* area, where Thunderbird is completely broken.
Sigh!

When
Mozilla declared it was considering dumping Thunderbird onto other open-source organizations (like how OpenOffice got dumped at the Apache Software Foundation) was when I decided to terminate my trial of
Thunderbird. Well, that and my exasperation with Thunderbird that
pushed me to also dump it after a 6-month trial.

Why did it take you *that* long! :-)

Out of interest: What are you using now (for e-mail)?

[...]
--- Synchronet 3.18a-Linux NewsLink 1.113

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Paul <nospam@needed.invalid> wrote:

VanguardLH wrote:

The more I have dig into Thunderbird and its bugs whether reported or
not, the more I get the feeling that the "developers" are CSCI
undergraduates, and over the years the turn over of volunteers resulted
in no old farts left that are intimate with the entire product.

I expect the team is smaller than the one in the photo here.

https://blog.thunderbird.net/2014/11/thunderbird-reorganizes-at-2014-toronto-summit/

21 people are shown in the photo, yet the article states:

"group of seven individuals were elected to comprise a Thunderbird
Council with the authority to make decisions affecting Thunderbird."

That article is dated back in 2014. Wonder what the attrition has been
since then.

"Thunderbird needs to have one or more full-time, paid staff to support shipping a stable, reliable product, and allow progress to be made on frequently-requested features. To this end, we plan to appeal directly
to our users for donations."

I doubt they've gotten enough donations sufficient for someone to accept
as variable minimum wage to be full-time paid staff. "Contributors"
sounds better than "volunteers".
--- Synchronet 3.18a-Linux NewsLink 1.113

From Newsgroup: comp.sys.ibm.pc.hardware.storage

On 5/12/2020 9:30 AM, Frank Slootweg wrote:

Nope, it's off there [1] as well.

I've been using Thunderbird for 10 years and there are no .wdseml
files, no .mozmsgs folders and the 'Allow Windows Search to search
messages' option is unticked in both [2] places.

[1] Where 'there' is as you mentioned:

Tools -> Options -> 'Advanced' major tab -> 'General' sub-tab -> System Integration -> Always check to see if Thunderbird is the default mail
client on startup -> Check Now.... -> In the 'System Integration' popup,
the 'Allow Windows Search to search messages' option is*not* ticked.

[2] It's of course [3] one option, which you can set or unset in these
two places.

[3] OTOH, considering this 'Mozilla' crap, 'of course' is a rather
tricky notion!:-(

I found that it was unchecked in base directory, but it was checked in
the pop-up dialog box. It's now unchecked in both places, and it's
stayed that way so far after a few restarts. I'll check again ... yup,
still unticked.

Yousuf Khan
--- Synchronet 3.18a-Linux NewsLink 1.113

From Newsgroup: comp.sys.ibm.pc.hardware.storage

Frank Slootweg <this@ddress.is.invalid> wrote:

Why did it take you *that* long [to abandon the trial of Tbird]! :-)

I had trialed Tbird several times, like perhaps once per year for
several years. I would also trial other e-mail clients (I did not need
nor want a combo email+newsgroups client). Often those lasted less than
a month for when I got pissed with Tbird enough to look elsewhere. For
the last trial, I was determined to use it long enough to get more
educated on how to use and configure it. After 6 months, I couldn't
stand it anymore.

Out of interest: What are you using now (for e-mail)?

I've used Outlook for a couple decades. I went to Office 365 for 3
years, but decided to quit the subscriptionware. Money was tighter back
then when I had to decide to renew or not. Although I switched to
LibreOffice, it has no e-mail program. I trialed a few candidates, but eventually settled on eM Client. It has lots of bugs most of which are
with its GUI, so nothing critical if not essential you tweak its GUI to
your likes.

I use the free version which limits me to only 2 accounts; however, it
will let you define more than 2 accounts, and then later you run into
problems polling those accounts, like ActiveSync stops working with
Hotmail forcing you to delete and recreate the account. The free
version does not restrict you from defining more than 2 accounts, but it
won't support it, and not only doesn't it support more than 2 accounts
but lets you define more and then fucks them up, even for IMAP accounts.
I got around the 2-account limitation in the free version by having one
of the monitored accounts poll other accounts, like configuring the
server-side options in Gmail and Hotmail to poll other accounts, so I
merged some accounts. If I needed more than 2 accounts in eM Client,
I'd buy it. I've found their developers/support do NOT visit their
forums, and peer support there pretty much devolves to just 1 attendant
there (who has no ability to escalate reported bugs to Dev). The author refuses to communicate with freeloaders on reporting bugs. You have to
buy to get any real support.

For a short time (a couple months), I trialed EssentialPIM. It also has
a 2-account limit for its free version, but I could work around that.
As I recall, back then they would watermark any printouts, but I think
they stopped that. However, to get ActiveSync (Exchange) support or
even CalDAV or other cloud-sync features meant having to buy it ($40 for
1-year support, $80 for lifetime license w/support). It's not subscriptionware, but what you pay dictates for how long you get
support. I do remember getting support from them despite I was using
their free version. https://www.essentialpim.com/pc-version/pro-vs-free
shows the differences. I remember I was close to buying ePIM, but don't
recall why I chose not. I do remember liking their Notes features in
the Pro version which made it similar to Microsoft's OneNote; however,
later Microsoft made OneNote free to everyone (and what I use), so that
lure fizzled.

At this point, and after using eM Client for just under 5 months (in
this latest trial since I trialed it a few times before), I might bite
the bullet and go back to the Office 365 subscriptionware to get Outlook
(plus LibreOffice Writer and Calc have been a little disappointing).
Rather than pay Microsoft's high subscription price of $99/year, I only
paid $33/year when I last used Office 365. So, I got 3 years for the
price of 1, and registering each subscription added it to the total subscription period, so I had 3 years of subscription before deciding
not to continue.

So, I'm on the fence right now. Do I continue using eM Client for free
with its buggy GUI with the 2-account limitation (which isn't enforced
but causes problems if you create more than 2 accounts)? Do I pay for
eM Client ($50 for a lifetime license, not subscriptionware), so I can
report the bugs to Dev (and not futilely in their forums) and hope they
get addressed? Do I get EssentialPIM Pro for $80 lifetime? Or do I buy
1-year licenses for Microsoft Office 365 (and cheaper from a reseller
instead of direct from Microsoft)?

I know some folks that just use the webmail client from the e-mail
provider (hotmail.com/outlook.com, gmail.com, comcast.net, etc). If you
want a local e-mail alert tool, there are lots of those (using POP or
IMAP). However, I do like the e-mail + calendar + contact integration
of eM Client, ePIM, and Outlook to allow the same local UI access to all
of those components across multiple hosts. You can get Google Chrome to
alert you to, of course, only Gmail new mails, but I don't leave the web browsers loaded all the time (I use them, and then exit them), plus
Chrome is my backup web browser while Firefox is my primary web browser.

I even tried the Mail, Calendar, and People apps from Microsoft that
comes bundled in Windows 10. To save my keyboard from repeated fist
banging, I quit using those.

I'm still old school in using local clients mostly because the web
clients are so dismally anemic. I want more than the majority of boobs,
er, users that are satisfied with less ... much less.
--- Synchronet 3.18a-Linux NewsLink 1.113