Hello All.

I made another patch slowly moving to rework charsets recoding and iconv.

I'd appreciate if someone may build it from branch 'Aliases' and try to navigate your messages and write new messages.

Especially if you have xlatcharsetalias entries in your config.

https://github.com/golded-plus/golded-plus/tree/Aliases

Vitaliy

--- GoldED+/LNX 1.1.5-b20231030
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-02-22 23:10:08, you wrote to All:

I made another patch slowly moving to rework charsets recoding and
iconv.

I'd appreciate if someone may build it from branch 'Aliases' and try to navigate your messages and write new messages.

I'm using it now. It seems to work just as well as my previous version GED+LNX 1.1.5-b20240209, from 'master', in my current config.

Especially if you have xlatcharsetalias entries in your config.

I don't use any of those though...

https://github.com/golded-plus/golded-plus/tree/Aliases

--- GoldED+/LNX 1.1.5-b20231030

You're not using it yourself? ;-)


Bye, Wilfred.

--- FMail-lnx64 2.2.1.1
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

23 Feb 24 12:31, you wrote to me:

I made another patch slowly moving to rework charsets recoding
and iconv.

I'd appreciate if someone may build it from branch 'Aliases' and
try to navigate your messages and write new messages.

I'm using it now. It seems to work just as well as my previous version GED+LNX 1.1.5-b20240209, from 'master', in my current config.

Sounds good. Thanks.

Especially if you have xlatcharsetalias entries in your config.

I don't use any of those though...

But you have xlatcharset in your config. That was changed too.

https://github.com/golded-plus/golded-plus/tree/Aliases

--- GoldED+/LNX 1.1.5-b20231030

You're not using it yourself? ;-)

I tested it in my development computer. Just didn't have time to update it on this one. Will do later today.

Vitaliy

--- GoldED+/LNX 1.1.5-b20231030
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-02-23 07:10:06, you wrote to me:

Especially if you have xlatcharsetalias entries in your config.

I don't use any of those though...

But you have xlatcharset in your config. That was changed too.

I don't have any in my standard configuration. But if I include charsets.cfg from the project. All messages in the stats area with linedrawing characters don't look ok. But that's the same in the previous version of golded that I was using.

Bye, Wilfred.

--- FMail-lnx64 2.2.1.1
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

23 Feb 24 16:31, you wrote to me:

Especially if you have xlatcharsetalias entries in your config.

I don't use any of those though...

But you have xlatcharset in your config. That was changed too.

I don't have any in my standard configuration. But if I include charsets.cfg from the project. All messages in the stats area with linedrawing characters don't look ok. But that's the same in the
previous version of golded that I was using.

If you don't have any - pseudo-graphics won't work unless you have console in same encoding as message.

Because GoldEd will only do charset translations if you have it configured. Otherwise it just prints message as is. And codes for pseudo-graphics is different in cp437 (message charset) and utf-8 (your terminal charset).

When you have charsets configured - there is another issue. GoldEd may convert from one byte encodings like cp437 to utf-8, but can't convert it back from utf-8 to cp437. I'm working on that. Have no idea, how much time it may take though.

Vitaliy

--- GoldED+/LNX 1.1.5-b20240223
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-02-25 19:06:26, you wrote to me:

I don't have any in my standard configuration. But if I include
charsets.cfg from the project. All messages in the stats area with
linedrawing characters don't look ok. But that's the same in the
previous version of golded that I was using.

If you don't have any - pseudo-graphics won't work unless you have console in same encoding as message.

Because GoldEd will only do charset translations if you have it configured.
Otherwise it just prints message as is. And codes for pseudo-graphics is different in cp437 (message charset) and utf-8 (your terminal charset).

It seems to work best for me this way. I never got the utf-8 terminal working correctly...

When you have charsets configured - there is another issue. GoldEd may convert from one byte encodings like cp437 to utf-8, but can't convert it back from utf-8 to cp437. I'm working on that. Have no idea, how much time it may take though.

I'll wait for that, before I have another go at it! ;-)

Bye, Wilfred.

--- FMail-lnx64 2.2.1.1
* Origin: FMail development HQ (2:280/464)

On Mon, 26 Feb 2024 01:06:26 -0700, Vitaliy Aksyonov -> Wilfred Van Velzen wrote:

Hello Wilfred.

23 Feb 24 16:31, you wrote to me:

 VA>>>>> Especially if you have xlatcharsetalias entries in your config.

 WvV>>>> I don't use any of those though...

 VA>>> But you have xlatcharset in your config. That was changed too.

 WvV>> I don't have any in my standard configuration. But if I include
 WvV>> charsets.cfg from the project. All messages in the stats area with
 WvV>> linedrawing characters don't look ok. But that's the same in the
 WvV>> previous version of golded that I was using.

If you don't have any - pseudo-graphics won't work unless you have console in same encoding as message.

This is why I wonder if he has a cp437 terminal setup, and this is why
these messages look fine to him. Whereas, I'm using a utf-8 terminal so I
have to translate everything to look ok.

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:115.0) Gecko/20100101 Thunderb
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

On Fri, 23 Feb 2024 05:10:08 -0700, Vitaliy Aksyonov -> All wrote:

I made another patch slowly moving to rework charsets recoding and iconv.

I'd appreciate if someone may build it from branch 'Aliases' and try to navigate your messages and write new messages.

Especially if you have xlatcharsetalias entries in your config.

https://github.com/golded-plus/golded-plus/tree/Aliases

When adding this patch does Golded need to be compiled with iconv support (ICONV=1)?

I would imagine Wilfred wouldn't see anything different if he's not using
any kind of character translations. But if one uses charsets.cfg this would
be a good test for you.

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:115.0) Gecko/20100101 Thunderb
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

Hello Nicholas.

26 Feb 24 16:50, you wrote to me:

?aVA>>>>>> Especially if you have xlatcharsetalias entries in your
?aVA>>>>>> config.

?aWvV>>>>> I don't use any of those though...

?aVA>>>> But you have xlatcharset in your config. That was changed
?aVA>>>> too.

?aWvV>>> I don't have any in my standard configuration. But if I
?aWvV>>> include charsets.cfg from the project. All messages in the
?aWvV>>> stats area with linedrawing characters don't look ok. But
?aWvV>>> that's the same in the previous version of golded that I was
?aWvV>>> using.

If you don't have any - pseudo-graphics won't work unless you
have console in same encoding as message.

This is why I wonder if he has a cp437 terminal setup, and this is why these messages look fine to him. Whereas, I'm using a utf-8 terminal
so I have to translate everything to look ok.

Yes. You need to use translation in this case. The problem is that GoldEd may translate from one byte charset to UTF-8, but not in reverse direction yet. That mean that you'll be able to read messages, but not write them. Actually, you'll be able to write them if no symbols with codes > 127 are used. But CHRS kludge in message will look like UTF-8 2.

Vitaliy

--- GoldED+/LNX 1.1.5-b20240223
* Origin: Aurora, Colorado (1:104/117)

Hello Nicholas.

26 Feb 24 17:11, you wrote to me:

I made another patch slowly moving to rework charsets recoding
and iconv.

I'd appreciate if someone may build it from branch 'Aliases' and
try to navigate your messages and write new messages.

Especially if you have xlatcharsetalias entries in your config.

https://github.com/golded-plus/golded-plus/tree/Aliases

When adding this patch does Golded need to be compiled with iconv
support (ICONV=1)?

No. Iconv support is far from completion yet.

I would imagine Wilfred wouldn't see anything different if he's not
using any kind of character translations. But if one uses charsets.cfg this would be a good test for you.

Yes. He shall not see any difference. And if he does - it mean I have a bug in my code. That's why I want as many people with different settings as possible to try it.

Vitaliy

... He ?o??o? e?u?o? ???? ?e?o?e?!
--- GoldED+/LNX 1.1.5-b20240223
* Origin: Aurora, Colorado (1:104/117)

On Tue, 27 Feb 2024 12:59:54 -0700, Vitaliy Aksyonov -> Nicholas Boel wrote:

This is why I wonder if he has a cp437 terminal setup, and this is why
these messages look fine to him. Whereas, I'm using a utf-8 terminal
so I have to translate everything to look ok.

Yes. You need to use translation in this case. The problem is that
GoldEd may translate from one byte charset to UTF-8, but not in reverse direction yet. That mean that you'll be able to read messages, but not write them. Actually, you'll be able to write them if no symbols with codes > 127 are used. But CHRS kludge in message will look like UTF-8 2.

Yes, I've known this for quite some time. And my current issue has to do
with reading messages (the one we've been using as an example) specifically translated from cp437 to utf-8.

Otherwise, if I absolutely have to reply to a message containing cp437 characters, I just don't quote those - as I know they won't be translated properly.

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:115.0) Gecko/20100101 Thunderb
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

Hello Vitaliy,

On Tuesday February 27 2024 07:02, you wrote to me:

Yes. He shall not see any difference. And if he does - it mean I have
a bug in my code. That's why I want as many people with different
settings as possible to try it.

Ok, I've installed your branch in a separate location as my other Golded install.

f5a1c8730f6dd5f8d56154822f68eb3a34f18c5a (b20231112) is what I'm currently using in my main directory, as that is the last version that works for me, just prior to you reverting those commits.

This version is, as far as I can tell, identical visually to the latest master branch - which doesn't work for me.

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- GoldED+/LNX 1.1.5-b20240223
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

Hello Nicholas.

27 Feb 24 17:23, you wrote to me:

On Tue, 27 Feb 2024 12:59:54 -0700, Vitaliy Aksyonov -> Nicholas Boel wrote:

This is why I wonder if he has a cp437 terminal setup, and this
is why these messages look fine to him. Whereas, I'm using a
utf-8 terminal so I have to translate everything to look ok.

Yes. You need to use translation in this case. The problem is
that GoldEd may translate from one byte charset to UTF-8, but not
in reverse direction yet. That mean that you'll be able to read
messages, but not write them. Actually, you'll be able to write
them if no symbols with codes > 127 are used. But CHRS kludge in
message will look like UTF-8 2.

Yes, I've known this for quite some time. And my current issue has to
do with reading messages (the one we've been using as an example) specifically translated from cp437 to utf-8.

Otherwise, if I absolutely have to reply to a message containing cp437 characters, I just don't quote those - as I know they won't be
translated properly.

I played with your configuration and have a good and bad news for you.

Good:
- I reproduced your issue.
- GoldEd correctly converts pseudo-graphics from cp437 to utf-8.

Bad:
- GoldEd does not support unicode. Even if you compile it with ncursesw, it still uses non unicode versions of functions to print text. That's why you see those escape sequences instead of pseudo-graphics symbols.

I still suggest you to use one-byte locale for GoldEd. And remember, you don't need to switch whole system to that locale, because in Linux locale is a property of a process. So you may have UTF-8 everywhere and cp437 for GoldEd. Most of terminals (including Putty) support different charsets.

Another option - is to use external editor, but that won't help you with message reader.

Vitaliy

--- GoldED+/LNX 1.1.5-b20240223
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-02-23 07:10:06, you wrote to me:

I made another patch slowly moving to rework charsets recoding
and iconv.

I'd appreciate if someone may build it from branch 'Aliases' and
try to navigate your messages and write new messages.

I'm using it now. It seems to work just as well as my previous
version GED+LNX 1.1.5-b20240209, from 'master', in my current config.

Sounds good. Thanks.

I switched back to the master branch because this patch was merged. Of course there are no changes in its operation for me.

I didn't mention this before, but there are some linker warnings:

../goldlib/uulib/libuulib.a(uunconc.cpp.o): In function `UUDecode(_uulist*)': uunconc.cpp:(.text+0x34e2): warning: the use of `tempnam' is dangerous, better use `mkstemp'
CMakeFiles/golded.dir/gedoit.cpp.o: In function `WriteMsgs(GMsg*)': gedoit.cpp:(.text+0xa51): warning: the use of `mktemp' is dangerous, better use `mkstemp'

They should be easy to address.


Bye, Wilfred.

--- FMail-lnx64 2.2.1.1
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

28 Feb 24 11:01, you wrote to me:

I didn't mention this before, but there are some linker warnings:

../goldlib/uulib/libuulib.a(uunconc.cpp.o): In function `UUDecode(_uulist*)': uunconc.cpp:(.text+0x34e2): warning: the use of `tempnam' is dangerous, better use
`mkstemp' CMakeFiles/golded.dir/gedoit.cpp.o: In function `WriteMsgs(GMsg*)': gedoit.cpp:(.text+0xa51): warning: the use of `mktemp' is dangerous, better use `mkstemp'

They should be easy to address.

I've seen those. I'm not sure that mkstemp is available in all systems GoldEd is possible to build.

Vitaliy

... O?u? ? ?o?e - xy?e ?a?apu?a...
--- GoldED+/LNX 1.1.5-b20240223
* Origin: Aurora, Colorado (1:104/117)

On Wed, 28 Feb 2024 05:53:06 -0700, Vitaliy Aksyonov -> Nicholas Boel wrote:

I played with your configuration and have a good and bad news for you.

Good:
- I reproduced your issue.
- GoldEd correctly converts pseudo-graphics from cp437 to utf-8.

Well, it did. Until you reverted those changes. :(

Bad:
- GoldEd does not support unicode. Even if you compile it with ncursesw, it still uses non unicode versions of functions to print text. That's
why you see those escape sequences instead of pseudo-graphics symbols.

Something happened recently where it made it quite a bit worse. I was
reading utf-8 Cyrillic, Greek, Japanese, Chinese, etc just fine in Golded
until the most recent version.

What changed with the ncurses init that was reverted?

And why does that change affect me opposite of a cp437 locale user?

If all I'm doing is translating from cp437 to utf-8 (or anything to utf-8)
I should still be able to read it properly, as I have been.. until
recently. Whatever you were doing with ncurses init helped me. I was able
to read utf-8 messages perfectly fine. Almost every single "Merry
Christmas" or "Happy New Year" Michiel posts yearly (except maybe a 2 or 3) were perfectly readable in Golded. The latest version they are not.

I still suggest you to use one-byte locale for GoldEd. And remember, you don't need to switch whole system to that locale, because in Linux
locale is a property of a process. So you may have UTF-8 everywhere and cp437 for GoldEd. Most of terminals (including Putty) support different charsets.

I'll pass on the suggestion :). I'll just keep using the last version that worked for me (minus a couple badly displayed ascii line characters, and
keep testing newer versions to see if I ever get the display back that I
lost. :)

Another option - is to use external editor, but that won't help you with message reader.

I already do this (I have always used nano with Golded). Viewing and
writing in an external editor has never been a problem. Until recently,
only reading became an issue. So the whole time it was working for me you actually broke something instead? :((

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:115.0) Gecko/20100101 Thunderb
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

On Wed, 28 Feb 2024 17:01:38 +0100, Wilfred Van Velzen -> Vitaliy Aksyonov wrote:

I switched back to the master branch because this patch was merged. Of course there are no changes in its operation for me.

So everything from the "Aliases" branch has been merged into master now?
Unless Vitaliy is going to continue doing things there for testing, I may
as well delete that install (I installed it separately as to not mess up
what is actually working for me).

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:115.0) Gecko/20100101 Thunderb
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

Hi Vitaliy,

On 28 Feb 24 11:03, Vitaliy Aksyonov wrote to Wilfred van Velzen:
about: "Re: Need volonteers to test another patch":

I didn't mention this before, but there are some linker warnings:

../goldlib/uulib/libuulib.a(uunconc.cpp.o): In function
`UUDecode(_uulist*)': uunconc.cpp:(.text+0x34e2): warning: the use of
`tempnam' is dangerous, better use
`mkstemp' CMakeFiles/golded.dir/gedoit.cpp.o: In function
`WriteMsgs(GMsg*)': gedoit.cpp:(.text+0xa51): warning: the use of
`mktemp' is dangerous, better use `mkstemp'

They should be easy to address.

I've seen those. I'm not sure that mkstemp is available in all systems GoldEd is possible to build.

mkstemp() is: 4.3BSD, POSIX.1-2001

If not available you could make your own from mktemp() and open() I suppose. I could find atleast 1 example on the net...


Wilfred.

--- FMail-W64 2.2.0.0
* Origin: point@work (2:280/464.112)

Hi Nicholas,

On 28 Feb 24 18:36, Nicholas Boel wrote to Wilfred van Velzen:
about: "Need volonteers to test another patch":

I switched back to the master branch because this patch was merged.
Of course there are no changes in its operation for me.

So everything from the "Aliases" branch has been merged into master now?

As far as I can tell from the git log: yes

https://paste.opensuse.org/pastes/8223c3c71052

Unless Vitaliy is going to continue doing things there for testing, I
may as well delete that install (I installed it separately as to not
mess up what is actually working for me).

Your choice! ;-)


Wilfred.

--- FMail-W64 2.2.0.0
* Origin: point@work (2:280/464.112)

Hello Nicholas.

28 Feb 24 18:33, you wrote to me:

I played with your configuration and have a good and bad news for
you.

Good:
- I reproduced your issue.
- GoldEd correctly converts pseudo-graphics from cp437 to utf-8.

Well, it did. Until you reverted those changes. :(

Most probably it was some combination which made it looking almost correct. I think that screen may be the reason you saw pseudo-graphics more or less correctly. Remember those line wraps? That happens because GoldEd converts those symbols to UTF-8 first. All pseudo-graphics symbols represented as 3 bytes. So that line become 3 times longer in bytes. Then GoldEd tries to split message to lines and it uses bytes! not symbols. That's why it splits the line in the middle of those pseudo-graphics. Even worse, it may tear apart one UTF-8 symbol to two lines and it will be displayed incorrectly.

GoldEd cannot work correctly with multibyte sequences. And even if it looks "correct", it's just because most English letters has same codes in cp437 and UTF-8.

If you want to keep using UTF-8, I may only suggest to find version, which "works" for you and stick to it.

Until full UTF-8 support implemented in GoldEd (if that ever happen), don't expect it to work correctly, sorry.

Bad:
- GoldEd does not support unicode. Even if you compile it with
ncursesw, it still uses non unicode versions of functions to
print text. That's why you see those escape sequences instead of
pseudo-graphics symbols.

Something happened recently where it made it quite a bit worse. I was reading utf-8 Cyrillic, Greek, Japanese, Chinese, etc just fine in
Golded until the most recent version.

I understand, that it might "work", but it was pure luck.

What changed with the ncurses init that was reverted?

First commit I did - changed order when ncurses initializes to be able to print some text, which doesn't use ncurses functions. That is used, when you run it with "-INSTALL". Before my first change with ncurses, that text was not displayed. I tried to fix that. But it caused some issues to dome sysops and what I did in recent commit - revert it back to what it was before commit 8e9f3518ac9b3b32676e7b7563e92cc44e7b5ba7.

And why does that change affect me opposite of a cp437 locale user?

Your setup is not supported and it's hard to say why it even works. Unfortunately I'm not big ncurses expert and hard to say what may go wrong.

If all I'm doing is translating from cp437 to utf-8 (or anything to
utf-8) I should still be able to read it properly, as I have been..
until recently. Whatever you were doing with ncurses init helped me. I
was able to read utf-8 messages perfectly fine. Almost every single
"Merry Christmas" or "Happy New Year" Michiel posts yearly (except
maybe a 2 or 3) were perfectly readable in Golded. The latest version
they are not.

Let me try to explain. Some UTF-8 characters have more than one byte. But GoldEd "prints" them to screen one by one. Also it applies some attributes. It uses non-unicode versions of ncurses functions and if that byte looks like special character to ncurses, it escapes it. That's why you see those weird character sequences with ~.

I still suggest you to use one-byte locale for GoldEd. And
remember, you don't need to switch whole system to that locale,
because in Linux locale is a property of a process. So you may
have UTF-8 everywhere and cp437 for GoldEd. Most of terminals
(including Putty) support different charsets.

I'll pass on the suggestion :). I'll just keep using the last version
that worked for me (minus a couple badly displayed ascii line
characters, and keep testing newer versions to see if I ever get the display back that I lost. :)

It's your choice. Just be aware, that if it works - it's just pure luck and don't expect it to last. Until we implement UTF-8 support. It may take years. Or never happen. It's not so easy to do it with backward compatibility wih all older systems like DOS or OS/2.

Another option - is to use external editor, but that won't help
you with message reader.

I already do this (I have always used nano with Golded). Viewing and writing in an external editor has never been a problem. Until
recently, only reading became an issue. So the whole time it was
working for me you actually broke something instead? :((

I'm sorry about that, but it's not much we can do right now.

Vitaliy

--- GoldED+/LNX 1.1.5-b20240223
* Origin: Aurora, Colorado (1:104/117)

Hello Nicholas.

28 Feb 24 18:36, you wrote to Wilfred van Velzen:

I switched back to the master branch because this patch was
merged. Of course there are no changes in its operation for me.

So everything from the "Aliases" branch has been merged into master
now? Unless Vitaliy is going to continue doing things there for
testing, I may as well delete that install (I installed it separately
as to not mess up what is actually working for me).

Aliases branch was not merged yet. It was another set of patches from Nil. I'll work on Aliases branch merge soon.

And I very appreciate your help with testing! GoldEd has too many settings to be able to test on my computer and even imagine different combinations.

And it doesn't have any unit tests, that doesn't help. :(

Vitaliy

--- GoldED+/LNX 1.1.5-b20240223
* Origin: Aurora, Colorado (1:104/117)

Hello Wilfred.

29 Feb 24 08:59, you wrote to me:

I didn't mention this before, but there are some linker
warnings:

../goldlib/uulib/libuulib.a(uunconc.cpp.o): In function
`UUDecode(_uulist*)': uunconc.cpp:(.text+0x34e2): warning: the
use of `tempnam' is dangerous, better use `mkstemp'
CMakeFiles/golded.dir/gedoit.cpp.o: In function
`WriteMsgs(GMsg*)': gedoit.cpp:(.text+0xa51): warning: the use
of `mktemp' is dangerous, better use `mkstemp'

They should be easy to address.

I've seen those. I'm not sure that mkstemp is available in all
systems GoldEd is possible to build.

mkstemp() is: 4.3BSD, POSIX.1-2001

If not available you could make your own from mktemp() and open() I suppose. I could find atleast 1 example on the net...

It's not the biggest issue now. But I'll think about it. Thanks.

Vitaliy

--- GoldED+/LNX 1.1.5-b20240223
* Origin: Aurora, Colorado (1:104/117)

Hello Vitaliy,

On Saturday March 02 2024 09:13, you wrote to me:

Most probably it was some combination which made it looking almost correct. I think that screen may be the reason you saw pseudo-graphics

I'm no longer using screen or tmux, and this version (along with all previous versions since you made the ncurses change) work the same and show UTF-8 properly, except for what you describe below:

more or less correctly. Remember those line wraps? That happens
because GoldEd converts those symbols to UTF-8 first. All
pseudo-graphics symbols represented as 3 bytes. So that line become 3 times longer in bytes. Then GoldEd tries to split message to lines and
it uses bytes! not symbols. That's why it splits the line in the
middle of those pseudo-graphics. Even worse, it may tear apart one
UTF-8 symbol to two lines and it will be displayed incorrectly.

Yeah, I've noticed most of this.. and thank you for your explanation. At least now I know why it is happening.

GoldEd cannot work correctly with multibyte sequences. And even if it looks "correct", it's just because most English letters has same codes
in cp437 and UTF-8.

Maybe simple ones, like german umlauts and whatnot. But cp437 doesn't have any Cyrillic, Greek, Japanese, Chinese, etc.

If you want to keep using UTF-8, I may only suggest to find version,
which "works" for you and stick to it.

I already have!

Until full UTF-8 support implemented in GoldEd (if that ever happen), don't expect it to work correctly, sorry.

That's ok. I had it somewhat working for awhile, the latest reverts have changed that. I don't have an issue going back to a "lucky" version. ;)

It's your choice. Just be aware, that if it works - it's just pure
luck and don't expect it to last. Until we implement UTF-8 support. It
may take years. Or never happen. It's not so easy to do it with
backward compatibility wih all older systems like DOS or OS/2.

I don't mind being lucky sometimes.

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- GoldED+/LNX 1.1.5-b20231112
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

Hello Nicholas.

02 Mar 24 14:44, you wrote to me:

Most probably it was some combination which made it looking
almost correct. I think that screen may be the reason you saw
pseudo-graphics

I'm no longer using screen or tmux, and this version (along with all previous versions since you made the ncurses change) work the same and show UTF-8 properly, except for what you describe below:

more or less correctly. Remember those line wraps? That happens
because GoldEd converts those symbols to UTF-8 first. All
pseudo-graphics symbols represented as 3 bytes. So that line
become 3 times longer in bytes. Then GoldEd tries to split
message to lines and it uses bytes! not symbols. That's why it
splits the line in the middle of those pseudo-graphics. Even
worse, it may tear apart one UTF-8 symbol to two lines and it
will be displayed incorrectly.

Yeah, I've noticed most of this.. and thank you for your explanation.
At least now I know why it is happening.

GoldEd cannot work correctly with multibyte sequences. And even
if it looks "correct", it's just because most English letters has
same codes in cp437 and UTF-8.

Maybe simple ones, like german umlauts and whatnot. But cp437 doesn't
have any Cyrillic, Greek, Japanese, Chinese, etc.

If you want to keep using UTF-8, I may only suggest to find
version, which "works" for you and stick to it.

I already have!

Until full UTF-8 support implemented in GoldEd (if that ever
happen), don't expect it to work correctly, sorry.

That's ok. I had it somewhat working for awhile, the latest reverts
have changed that. I don't have an issue going back to a "lucky"
version. ;)

It's your choice. Just be aware, that if it works - it's just
pure luck and don't expect it to last. Until we implement UTF-8
support. It may take years. Or never happen. It's not so easy to
do it with backward compatibility wih all older systems like DOS
or OS/2.

I don't mind being lucky sometimes.

I played little bit more with the code and different settings and found what was the difference between those two versions.

First - you use "wide" ncurses - ncursesw. I use ncurses.
Second. GoldEd incorrectly initializes ncurses in reverted version (like it was before I started to make any changes in GoldEd code). ncurses documentation explicitly says that before call initscr(), locale has to be set with setlocale. Otherwise it will use incorrect settings. Most probably plain C locale. I kinda get picture very similar to yours.

Did you have an issue with non-English chars when scrolling the message? What I see in UFT-8 terminal now - most of unicode text displayed correctly and pseudo-graphic too. Only pseudo-graphics lines broken similar to yours. And when I scroll text down - it corrupts.

If you may try to add one line of code to latest master and try it - that would be helpful.

goldlib/gcui/gkbdbase.cpp
line 149, right before initscr add this:

setlocale(LC_ALL, "");

If that gives you expected result - I'll push this change to master.

Hope you still want to dig this. :)

Vitaliy

--- GoldED+/LNX 1.1.5-b20240223
* Origin: Aurora, Colorado (1:104/117)

On Sun, 3 Mar 2024 04:42:52 -0700, Vitaliy Aksyonov -> Nicholas Boel wrote:

I played little bit more with the code and different settings and found what was the difference between those two versions.

First - you use "wide" ncurses - ncursesw. I use ncurses.
Second. GoldEd incorrectly initializes ncurses in reverted version (like it was before I started to make any changes in GoldEd code). ncurses documentation explicitly says that before call initscr(), locale has to
be set with setlocale. Otherwise it will use incorrect settings. Most probably plain C locale. I kinda get picture very similar to yours.

For the record, I don't compile golded with WIDE_NCURSE=1. I have tried
it in the past, but I don't see much, if any difference. However, I have
also realized that in my distro (Archlinux) I don't have to install two separate packages for that, either. ncurses comes with ncursesw
included. Is that maybe the difference?

Did you have an issue with non-English chars when scrolling the message?

Non-english characters usually display OK, and sometimes when you scroll
the message, some of them (usually moreso the line and block drawings
converted from cp437) will gain a red background with green "^"
characters) or as you say below, corrupts. Using page down instead of
line scrolling keeps the characters in tact, though.

What I see in UFT-8 terminal now - most of unicode text displayed correctly and pseudo-graphic too. Only pseudo-graphics lines broken similar to yours. And when I scroll text down - it corrupts.

Yes, I see this too. However, try using page down instead of the down
arrow key. Maybe that will point you in a direction as to how both key
presses are handled and you may be able to fix that as well!

As for the pseudo-graphics wrapped to the next line, I have a (probably
dumb) question about this: If the pseudo graphics were originally cp437
(single byte) and translated to utf-8, once they are translated are they
now multiple bytes per character?

If "UTF-8 uses 1 to 4 bytes to encode a single character", I guess what
I'm wondering is if the character was 1 byte to begin with, why wouldn't
it stay 1 byte when translated to utf-8? Or is it because those
_specific_ characters when in utf-8 are already multiple bytes?

If you may try to add one line of code to latest master and try it -
that would be helpful.

goldlib/gcui/gkbdbase.cpp
line 149, right before initscr add this:

setlocale(LC_ALL, "");

If that gives you expected result - I'll push this change to master.

It worked! Or at least it's back to the way it was, which is a good thing!

Hope you still want to dig this. :)

Definitely! I appreciate you willing to help, even though we both know
it's broken in more ways than one. I do understand that it is not
currently (and has never been) supported, but if we can make it "kind
of" work while not affecting others, I'm all for it!

I would rather have a somewhat-working program than a non-working
program. :)

THANK YOU!

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:115.0) Gecko/20100101 Thunderb
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

Hello Nicholas,

On Sunday March 03 2024 08:46, you wrote to Vitaliy Aksyonov:

As for the pseudo-graphics wrapped to the next line, I have a
(probably dumb) question about this: If the pseudo graphics were originally cp437 (single byte) and translated to utf-8, once they are translated are they now multiple bytes per character?

I prefer dumb quetion, they are easier to answer... ;-)

Yes, they are translated to multi (usually two for most characters used in Fidonet) byte characters. Only the ASCII characters (0-127) are not translated and so remain one byte.

If "UTF-8 uses 1 to 4 bytes to encode a single character", I guess
what I'm wondering is if the character was 1 byte to begin with, why wouldn't it stay 1 byte when translated to utf-8? Or is it because
those _specific_ characters when in utf-8 are already multiple bytes?

A non ASCII character can not be translated to one byte for the simple reason that the remaning 128 bytes with the highest bit set are not enough to encode ALL the characters in ALL the single byte characters sets. The whole idea of unicode is to encode ALL the characters of ALL those characters sets, CP437, CP850, CP 866, CP 1250, etc into ONE encoding scheme. One byte is just not enough for all.

To put it simple: if you want to encode CP437 and CP866, you could put CP437 OR CP866 in the first byte, but you need at least one bit more information which one it is; CP437 or CP866. That is not exactly how UTF-8 works but it should give you an idea of why just one byte can not be enough.


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20170303
* Origin: Nieuw Schnøørd (2:280/5555)

On Sun, 3 Mar 2024 22:45:34 +0100, Michiel Van Der Vlist -> Nicholas
Boel wrote:

MvdV> Yes, they are translated to multi (usually two for most characters used
MvdV> in Fidonet) byte characters. Only the ASCII characters (0-127) are not
MvdV> translated and so remain one byte.

Thanks for the explanation. While reading this, I did check the ASCII
table and the characters I'm referring to are all above 127. This also
is kind of reflected while using Golded, if I widen my screen more than
160 characters, less of those lines are wrapped to the next line.

However, it doesn't seem like I can widen my window enough to keep them
all on one line, so I'm guessing when they are translated to utf-8 they
are more than 2 bytes, since I'm well over double the width of an 80
character screen - which the original stat message was made for.

So, at this point it's basically working and displaying properly, but
then comes in the 'characters' vs 'bytes' thing that Golded isn't
supporting, so it is wrapping what it thinks is double, triple, or even quadruple the amount of 'characters' that are there.

MvdV> To put it simple: if you want to encode CP437 and CP866, you could put
MvdV> CP437 OR CP866 in the first byte, but you need at least one bit more
MvdV> information which one it is; CP437 or CP866. That is not exactly how
MvdV> UTF-8 works but it should give you an idea of why just one byte can not
MvdV> be enough.

Thank you for the explanation. This definitely helps me to understand
what is happening.

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:115.0) Gecko/20100101 Thunderb
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

Hello Nicholas,

On Sunday March 03 2024 10:33, you wrote to me:

So, at this point it's basically working and displaying properly, but
then comes in the 'characters' vs 'bytes' thing that Golded isn't supporting, so it is wrapping what it thinks is double, triple, or
even quadruple the amount of 'characters' that are there.

When it comes to mapping the number of characters to the number of bytes, when you look at the UTF-8 encoding table, about two screens down here:

https://en.wikipedia.org/wiki/UTF-8

You can see that the length of the byte sequence can be determined just by looking at the first byte. Look from bit 7 to the right. The number of '1' bits equals the number of bytes in the character. All the follow up bytes start with '10'. So to get the number of characters ignore the bytes starting with '10' when counting the bytes.

Breaking a line should only occur /before/ a byte starting with '0' or '11'.

Knowing all that it should be doable to let Golded display properly.

Perhaps the best strategy is to have Golded alway use UTF-8 internally. Almost everyone else does these days...

Thank you for the explanation. This definitely helps me to understand
what is happening.

You'r welcome.


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20170303
* Origin: Nieuw Schnøørd (2:280/5555)

Hello Nicholas.

03 Mar 24 10:33, you wrote to Michiel van der Vlist:

MvdV>> Yes, they are translated to multi (usually two for most
MvdV>> characters used in Fidonet) byte characters. Only the ASCII
MvdV>> characters (0-127) are not translated and so remain one byte.

Thanks for the explanation. While reading this, I did check the ASCII table and the characters I'm referring to are all above 127. This also
is kind of reflected while using Golded, if I widen my screen more
than 160 characters, less of those lines are wrapped to the next line.

However, it doesn't seem like I can widen my window enough to keep
them all on one line, so I'm guessing when they are translated to
utf-8 they are more than 2 bytes, since I'm well over double the width
of an 80 character screen - which the original stat message was made
for.

Pseudo-graphics symbols encoded to three bytes in UTF-8. That's why it's not enough.
So you either ignore it or make windows at least 3 times wider than that message. It might sound stupid and overkill, but it shall do the trick. :)

So, at this point it's basically working and displaying properly, but
then comes in the 'characters' vs 'bytes' thing that Golded isn't supporting, so it is wrapping what it thinks is double, triple, or
even quadruple the amount of 'characters' that are there.

MvdV>> To put it simple: if you want to encode CP437 and CP866, you
MvdV>> could put CP437 OR CP866 in the first byte, but you need at
MvdV>> least one bit more information which one it is; CP437 or CP866.
MvdV>> That is not exactly how UTF-8 works but it should give you an
MvdV>> idea of why just one byte can not be enough.

Thank you for the explanation. This definitely helps me to understand
what is happening.

Unicode is very complex. It's event worse than you may think. For example, some displayed symbols take one char space on screen, but others two.
Also one letter is not necessarily one Unicode "symbol". Also it may have non-printable symbols and many more.

Vitaliy

... C????o ?a ??epa??ee, ?o ?e ?o??? ?epe? ?e?...
--- GoldED+/LNX 1.1.5-b20240223
* Origin: Aurora, Colorado (1:104/117)

Hello Michiel.

03 Mar 24 22:31, you wrote to Nicholas Boel:

So, at this point it's basically working and displaying properly,
but then comes in the 'characters' vs 'bytes' thing that Golded
isn't supporting, so it is wrapping what it thinks is double,
triple, or even quadruple the amount of 'characters' that are
there.

MvdV> When it comes to mapping the number of characters to the number of
MvdV> bytes, when you look at the UTF-8 encoding table, about two screens
MvdV> down here:

MvdV> https://en.wikipedia.org/wiki/UTF-8

MvdV> You can see that the length of the byte sequence can be determined
MvdV> just by looking at the first byte. Look from bit 7 to the right. The
MvdV> number of '1' bits equals the number of bytes in the character. All
MvdV> the follow up bytes start with '10'. So to get the number of
MvdV> characters ignore the bytes starting with '10' when counting the
MvdV> bytes.

MvdV> Breaking a line should only occur /before/ a byte starting with '0' or
MvdV> '11'.

MvdV> Knowing all that it should be doable to let Golded display properly.

MvdV> Perhaps the best strategy is to have Golded alway use UTF-8
MvdV> internally. Almost everyone else does these days...

That would be perfect. It only takes huge amount of effort. Especially with keeping code backward compatible with systems, which may not have Unicode support. I keep thinking about it and looking for possible ways to implement.

For now I'd be happy to make iconv work properly. In this case GoldEd user may get rid of most (if not all) of translation tables. The problem is that source code has huge amounts of duplicated code and sometimes functions and variables names say nothing about what do they do. I spend huge amount of time just to understand what's going on.

For example function, which splits message to lines is almost 1000 lines long! It has variables, used in multiple places, it not only splits the message, but guess charset, do recoding and other fun stuff.

Vitaliy

--- GoldED+/LNX 1.1.5-b20240223
* Origin: Aurora, Colorado (1:104/117)

On Sun, 3 Mar 2024 22:28:22 -0700, Vitaliy Aksyonov -> Nicholas Boel wrote:

Pseudo-graphics symbols encoded to three bytes in UTF-8. That's why it's not enough.
So you either ignore it or make windows at least 3 times wider than that message. It might sound stupid and overkill, but it shall do the trick. :)

I usually just ignore it. But it would be nice if one day it displayed properly. :)

By the way, I see you added the Aliases branch to master, were you still planning on setting the locale before initscr?

I've updated my Golded to the latest, and then added that line again and recompiled. Still seems to be working as expected!

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:115.0) Gecko/20100101 Thunderb
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

Hello Vitaliy,

On Sunday March 03 2024 16:28, you wrote to Nicholas Boel:

Unicode is very complex. It's event worse than you may think. For
example, some displayed symbols take one char space on screen, but
others two. Also one letter is not necessarily one Unicode "symbol".
Also it may have non-printable symbols and many more.

True but we do not need a full 100% implementation for Fidonet. We can have a working implementation and ignore these details for now.


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20170303
* Origin: Nieuw Schnøørd (2:280/5555)

Hello Vitaliy,

On Sunday March 03 2024 16:31, you wrote to me:

MvdV>> Perhaps the best strategy is to have Golded alway use UTF-8
MvdV>> internally. Almost everyone else does these days...

That would be perfect. It only takes huge amount of effort. Especially with keeping code backward compatible with systems, which may not have Unicode support. I keep thinking about it and looking for possible
ways to implement.

Backwards compatibility is nice but there always comes a point that it gets in the way of progress and it has to be dropped. Are you thinking about the DOS version? If so I say, forget about it. Freeze the DOS version, the small minority that still uses DOS will have to make do with what they have fo the rest of the life of DOS.

Another way may be to not use UTF-8 internally but use two byte widechrs everywhere and simple store the raw unicode code point. Conversion to and from code point to UTF-8 is simple. That will limit the use to the first 65535 code points, but that might be enough for the remaining life of Fidonet. OTOH, that is almost the same as what Window XP did. It used UTF-16 internally and Microsoft now regrets that.

For example function, which splits message to lines is almost 1000
lines long! It has variables, used in multiple places, it not only
splits the message, but guess charset, do recoding and other fun
stuff.

Wauw!


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20170303
* Origin: Nieuw Schnøørd (2:280/5555)

Hello Nicholas.

03 Mar 24 19:53, you wrote to me:

Pseudo-graphics symbols encoded to three bytes in UTF-8. That's
why it's not enough. So you either ignore it or make windows at
least 3 times wider than that message. It might sound stupid and
overkill, but it shall do the trick. :)

I usually just ignore it. But it would be nice if one day it displayed properly. :)

By the way, I see you added the Aliases branch to master, were you
still planning on setting the locale before initscr?

That's the plan.

I've updated my Golded to the latest, and then added that line again
and recompiled. Still seems to be working as expected!

Great. Thanks for help!

Vitaliy

--- GoldED+/LNX 1.1.5-b20240223
* Origin: Aurora, Colorado (1:104/117)

Hello Michiel.

04 Mar 24 08:42, you wrote to me:

MvdV>>> Perhaps the best strategy is to have Golded alway use UTF-8
MvdV>>> internally. Almost everyone else does these days...

That would be perfect. It only takes huge amount of effort.
Especially with keeping code backward compatible with systems,
which may not have Unicode support. I keep thinking about it and
looking for possible ways to implement.

MvdV> Backwards compatibility is nice but there always comes a point that it
MvdV> gets in the way of progress and it has to be dropped. Are you thinking
MvdV> about the DOS version? If so I say, forget about it. Freeze the DOS
MvdV> version, the small minority that still uses DOS will have to make do
MvdV> with what they have fo the rest of the life of DOS.

We've been thinking about that option.

MvdV> Another way may be to not use UTF-8 internally but use two byte
MvdV> widechrs everywhere and simple store the raw unicode code point.
MvdV> Conversion to and from code point to UTF-8 is simple. That will limit
MvdV> the use to the first 65535 code points, but that might be enough for
MvdV> the remaining life of Fidonet. OTOH, that is almost the same as what
MvdV> Window XP did. It used UTF-16 internally and Microsoft now regrets
MvdV> that.

Best possible way is to use UTF-8 for all strings inside and only convert text when read/write from/to message base and to screen. And even if drop DOS support - need to take into account OS specifics for Unicode. As long as GoldEd uses fixed size buffers in many places - that's huge refactoring. Better to replace it with std::string almost everywhere.

For example function, which splits message to lines is almost
1000 lines long! It has variables, used in multiple places, it
not only splits the message, but guess charset, do recoding and
other fun stuff.

MvdV> Wauw!

That's one of the reasons, why progress is slow.

Vitaliy

--- GoldED+/LNX 1.1.5-b20240223
* Origin: Aurora, Colorado (1:104/117)

Hello Vitaliy,

On Monday March 04 2024 07:56, you wrote to me:

I've updated my Golded to the latest, and then added that line
again and recompiled. Still seems to be working as expected!

Great. Thanks for help!

Since I would just sit there like a deer in headlights if I were to jump into the code myself, I'll leave that up to you and just test where I can. :)

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- GoldED+/LNX 1.1.5-b20240302
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

Hello Vitaliy,

On Mon, 04 Mar 2024 07:56:56 -0700, you wrote to me:

Definitely not going to stop traffic or anything, but if you take a look at the above quote header line taken from golded.tpl, it seems when trying to use "@otzoffset" in golded.tpl it adds an extra space before it. I'm specifically using "On @odate @otzoffset," with only one space in between the two tokens, and the date I have set in gedlngus.cfg doesn't have a space at the end of it, either.

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- GoldED+/LNX 1.1.5-b20240302
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

Hello Nicholas.

04 Mar 24 18:29, you wrote to me:

I've updated my Golded to the latest, and then added that line
again and recompiled. Still seems to be working as expected!

Great. Thanks for help!

Since I would just sit there like a deer in headlights if I were to
jump into the code myself, I'll leave that up to you and just test
where I can. :)

I'll work on that. Wait couple of days please.

Vitaliy

--- GoldED+/LNX 1.1.5-b20240223
* Origin: Aurora, Colorado (1:104/117)

Hello Nicholas.

04 Mar 24 19:18, you wrote to me:

Definitely not going to stop traffic or anything, but if you take a
look at the above quote header line taken from golded.tpl, it seems
when trying to use "@otzoffset" in golded.tpl it adds an extra space before it. I'm specifically using "On @odate @otzoffset," with only
one space in between the two tokens, and the date I have set in gedlngus.cfg doesn't have a space at the end of it, either.

That shall be supersimple. I'll take a look on that too when have some time.

Vitaliy

--- GoldED+/LNX 1.1.5-b20240223
* Origin: Aurora, Colorado (1:104/117)

On Tue, 5 Mar 2024 03:20:26 -0700, Vitaliy Aksyonov -> Nicholas Boel wrote:

Definitely not going to stop traffic or anything, but if you take a
look at the above quote header line taken from golded.tpl, it seems
when trying to use "@otzoffset" in golded.tpl it adds an extra space
before it. I'm specifically using "On @odate @otzoffset," with only
one space in between the two tokens, and the date I have set in
gedlngus.cfg doesn't have a space at the end of it, either.

That shall be supersimple. I'll take a look on that too when have some time.

Thank you!

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:115.0) Gecko/20100101 Thunderb
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

Hello Vitaliy.

04 Mar 24 21:19, I wrote to Nicholas Boel:

I've updated my Golded to the latest, and then added that line
again and recompiled. Still seems to be working as expected!

Great. Thanks for help!

Since I would just sit there like a deer in headlights if I were
to jump into the code myself, I'll leave that up to you and just
test where I can. :)

I'll work on that. Wait couple of days please.

Pull request is sent. Wait for commit.

Unfortunately it will break pseudo-graphics in some other systems. But my change shall be there anyway. Ncurses shall not be initialized before setlocale().

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240223
* Origin: Aurora, Colorado (1:104/117)

Hello Nicholas.

05 Mar 24 20:27, you wrote to me:

Definitely not going to stop traffic or anything, but if you
take a look at the above quote header line taken from
golded.tpl, it seems when trying to use "@otzoffset" in
golded.tpl it adds an extra space before it. I'm specifically
using "On @odate @otzoffset," with only one space in between the
two tokens, and the date I have set in gedlngus.cfg doesn't have
a space at the end of it, either.

That shall be supersimple. I'll take a look on that too when have
some time.

Thank you!

This one sent for review as well.

PS. Take a look on my tearline. Now I'm using my latest changes. ;)

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hello Vitaliy,

On Monday March 04 2024 07:58, you wrote to me:

Best possible way is to use UTF-8 for all strings inside and only
convert text when read/write from/to message base and to screen.

I agree. That will be the easiest way to make as many Fidonet participants use UTF-8 all the way. With the sceen set to CP65001 writing to and from the screen should need no conversion.

And even if drop DOS support - need to take into account OS specifics
for Unicode.

Such as? Even OS/2 has full UTF-8 support doesn't it?

As long as GoldEd uses fixed size buffers in many places -
that's huge refactoring. Better to replace it with std::string almost everywhere.

Perhaps, but that won't solve the problem that when writing back to the message base strings have to be of fixed lenght for the To:, From:, Subj: and other fields. It may be necessary to truncate in order to fit. Truncating should be done on a UTF-8 sequence boundery. If need be step back until a byte with bit 7 and 6 set.

For example function, which splits message to lines is almost
1000 lines long! It has variables, used in multiple places, it
not only splits the message, but guess charset, do recoding and
other fun stuff.

MvdV>> Wauw!

That's one of the reasons, why progress is slow.

Keep up the good work!


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20170303
* Origin: Nieuw Schnøørd (2:280/5555)

Hello Michiel.

06 Mar 24 13:38, you wrote to me:

Best possible way is to use UTF-8 for all strings inside and only
convert text when read/write from/to message base and to screen.

MvdV> I agree. That will be the easiest way to make as many Fidonet
MvdV> participants use UTF-8 all the way. With the sceen set to CP65001
MvdV> writing to and from the screen should need no conversion.

And even if drop DOS support - need to take into account OS
specifics for Unicode.

MvdV> Such as? Even OS/2 has full UTF-8 support doesn't it?

Windows and Linux works with console very differently. Even support those two together is a challenge. Code is not very well encapsulated in GoldEd and OS specifics just everywhere.

I haven't tried OS/2 at all, so have no idea, how UTF-8 works there. Does it use ncurses too?

As long as GoldEd uses fixed size buffers in many places -
that's huge refactoring. Better to replace it with std::string
almost everywhere.

MvdV> Perhaps, but that won't solve the problem that when writing back to
MvdV> the message base strings have to be of fixed lenght for the To:,
MvdV> From:, Subj: and other fields. It may be necessary to truncate in
MvdV> order to fit. Truncating should be done on a UTF-8 sequence boundery.
MvdV> If need be step back until a byte with bit 7 and 6 set.

For example function, which splits message to lines is almost
1000 lines long! It has variables, used in multiple places, it
not only splits the message, but guess charset, do recoding and
other fun stuff.

MvdV>>> Wauw!

That's one of the reasons, why progress is slow.

MvdV> Keep up the good work!

Thanks. I try to fix or enhance something which doesn't break existing functionality. Hope some day we'll have UTF-8 support too.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-02 22:42:52, you wrote to Nicholas Boel:

If you may try to add one line of code to latest master and try it -
that would be helpful.

goldlib/gcui/gkbdbase.cpp
line 149, right before initscr add this:

setlocale(LC_ALL, "");

I pulled in the latest code with this change, and this breaks my configuration again. :-(

So I reverted that change, and everything is back to normal...


Bye, Wilfred.

--- FMail-lnx64 2.2.1.1
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

06 Mar 24 22:36, you wrote to me:

If you may try to add one line of code to latest master and try
it - that would be helpful.

goldlib/gcui/gkbdbase.cpp
line 149, right before initscr add this:

setlocale(LC_ALL, "");

I pulled in the latest code with this change, and this breaks my configuration again. :-(

So I reverted that change, and everything is back to normal...

One of the changes is very similar to that one which was reverted previously.

Remind me your setup please. Do you use UTF-8 locale? What terminal do you use? TERM variable.

Do you use local console or remote/ssh?


Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hello Vitaliy,

On Wednesday March 06 2024 10:32, you wrote to me:

I haven't tried OS/2 at all, so have no idea, how UTF-8 works there.
Does it use ncurses too?

I don't know. I tried OS/2 some 25-30 years ago and I liked it. But... I could not get it to work with Novell (Personal) Netware and so I dropped it. That problem was fixed later, but for me goodbye is goodbye. Well, most of the time...

Also OS/2 does not support IPv6 and probably never will, so that is another reason to let it rest in peace.


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20170303
* Origin: Nieuw Schnøørd (2:280/5555)

On Wed, 6 Mar 2024 03:53:14 -0700, Vitaliy Aksyonov -> Vitaliy Aksyonov
wrote:

Pull request is sent. Wait for commit.

Unfortunately it will break pseudo-graphics in some other systems. But
my change shall be there anyway. Ncurses shall not be initialized before setlocale().

Thank you, I'll update when it's pushed!

Why would it break pseudo-graphics in other systems? Or are you just
referring to other systems that don't actually use translation tables? :)

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:115.0) Gecko/20100101 Thunderb
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

On Thu, 7 Mar 2024 04:36:10 +0100, Wilfred Van Velzen -> Vitaliy
Aksyonov wrote:

I pulled in the latest code with this change, and this breaks my configuration again. :-(

So I reverted that change, and everything is back to normal...

Not 100% sure on this, but maybe you were just one of the lucky few that
never had to use translation tables. However, they should probably be
used no matter what kind of terminal or setup you have.

Nothing fancy. Just try including charsets.cfg, mess with 'xlatimport'
and 'xlatlocalset' a bit and see what happens.

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:115.0) Gecko/20100101 Thunderb
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

Hello Vitaliy,

On Tue, 05 Mar 2024 21:53:14 -0700, you wrote to you:

Pull request is sent. Wait for commit.

Unfortunately it will break pseudo-graphics in some other systems. But
my change shall be there anyway. Ncurses shall not be initialized
before setlocale().

The specific things you did for me are working as expected. Thank you!

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- GoldED+/LNX 1.1.5-b20240306
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

Windows and Linux works with console very differently. Even support
those two together is a challenge. Code is not very well encapsulated in GoldEd and OS specifics just everywhere.

duno if it'd be useful but nftp (ayukov.com) had a little ui unit for win/lin/os2/etc .. if only to see other examples

--- Mystic BBS v1.12 A47 2021/12/25 (Windows/32)
* Origin: cold fusion - cfbbs.net - grand rapids, mi (1:120/616)

Hello Nicholas.

06 Mar 24 17:06, you wrote to me:

Pull request is sent. Wait for commit.

Unfortunately it will break pseudo-graphics in some other
systems. But my change shall be there anyway. Ncurses shall not
be initialized before setlocale().

Thank you, I'll update when it's pushed!

Why would it break pseudo-graphics in other systems? Or are you just referring to other systems that don't actually use translation tables?
:)

It's not about translation tables, but about displaying the text. Ncurses has special logic to output pseudo-graphics and it most probably won't work correctly in Unicode locale. It was kinda "working" just because ncurses "thought" that it worked in "legacy" mode, not unicode.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hello Nicholas.

06 Mar 24 17:23, you wrote to me:

Pull request is sent. Wait for commit.

Unfortunately it will break pseudo-graphics in some other
systems. But my change shall be there anyway. Ncurses shall not
be initialized before setlocale().

The specific things you did for me are working as expected. Thank you!

Glad it works for you. And anyway my change was necessary, because ncurses was not initialized properly before.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hello fusion.

06 Mar 24 18:58, you wrote to me:

Windows and Linux works with console very differently. Even
support those two together is a challenge. Code is not very well
encapsulated in GoldEd and OS specifics just everywhere.

duno if it'd be useful but nftp (ayukov.com) had a little ui unit for win/lin/os2/etc .. if only to see other examples

Golded's editor has some FTN and own specifics, it's not just plain text editor. And not only editor a problem. GoldEd full of string manipulations. For example when it works with messages templates. Or uses clipboard. That's why while use some other editor with Unicode support won't solve all issues. Whole system need to be updated for Unicode.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-06 14:59:02, you wrote to me:

If you may try to add one line of code to latest master and try
it - that would be helpful.

goldlib/gcui/gkbdbase.cpp
line 149, right before initscr add this:

setlocale(LC_ALL, "");

I pulled in the latest code with this change, and this breaks my
configuration again. :-(

So I reverted that change, and everything is back to normal...

One of the changes is very similar to that one which was reverted previously.

Remind me your setup please. Do you use UTF-8 locale?

# sudo -u fido locale
LANG=POSIX
LC_CTYPE=en_US.UTF-8
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

(I also start golded with 'sudo -u fido')

But for my golded terminal in Konsole, I have the "Default character encoding" set to IBM850.

What terminal do you use? TERM variable.

# sudo -u fido echo $TERM
xterm

Do you use local console or remote/ssh?

Both, but this test I only did in my local console.


Bye, Wilfred.

--- FMail-lnx64 2.2.1.1
* Origin: FMail development HQ (2:280/464)

On Thu, 7 Mar 2024 13:02:18 -0700, Vitaliy Aksyonov -> Nicholas Boel wrote:

Why would it break pseudo-graphics in other systems? Or are you just
referring to other systems that don't actually use translation tables?
:)

It's not about translation tables, but about displaying the text.
Ncurses has special logic to output pseudo-graphics and it most probably won't work correctly in Unicode locale. It was kinda "working" just because ncurses "thought" that it worked in "legacy" mode, not unicode.

But, I'm the one using a unicode locale, and it is working. It seems to
only affect people using a cp437/cp850-style single bit locale.

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:115.0) Gecko/20100101 Thunderb
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

Hello Nicholas.

07 Mar 24 17:09, you wrote to me:

Why would it break pseudo-graphics in other systems? Or are you
just referring to other systems that don't actually use
translation tables?
:)

It's not about translation tables, but about displaying the text.
Ncurses has special logic to output pseudo-graphics and it most
probably won't work correctly in Unicode locale. It was kinda
"working" just because ncurses "thought" that it worked in
"legacy" mode, not unicode.

But, I'm the one using a unicode locale, and it is working. It seems
to only affect people using a cp437/cp850-style single bit locale.

For you it's "working". For him it shall work flawlessly. Answered in separate message.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hello Wilfred.

07 Mar 24 18:55, you wrote to me:

If you may try to add one line of code to latest master and try
it - that would be helpful.

goldlib/gcui/gkbdbase.cpp
line 149, right before initscr add this:

setlocale(LC_ALL, "");

I pulled in the latest code with this change, and this breaks
my configuration again. :-(

So I reverted that change, and everything is back to normal...

One of the changes is very similar to that one which was reverted
previously.

Remind me your setup please. Do you use UTF-8 locale?

# sudo -u fido locale
LANG=POSIX
LC_CTYPE=en_US.UTF-8
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

(I also start golded with 'sudo -u fido')

But for my golded terminal in Konsole, I have the "Default character encoding" set to IBM850.

OK. Your terminal works in IBM850, but you run golded may try to run in UTF-8. That won't work and it worked before just because golded was not initializing correctly!

Question. What is your XLatLocalSet in GoldEd?

What terminal do you use? TERM variable.

# sudo -u fido echo $TERM
xterm

Do you use local console or remote/ssh?

Both, but this test I only did in my local console.

I don't expect any difference, but it may be some.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-07 17:30:02, you wrote to me:

OK. Your terminal works in IBM850, but you run golded may try to run
in UTF-8. That won't work and it worked before just because golded was
not initializing correctly!

Ok, so how do I fix it? ;)

Question. What is your XLatLocalSet in GoldEd?

I don't have any.


Bye, Wilfred.

--- FMail-lnx64 2.2.1.1
* Origin: FMail development HQ (2:280/464)

Hi Vitaliy,

On 2024-03-08 09:02:51, I wrote to you:

OK. Your terminal works in IBM850, but you run golded may try to run
in UTF-8. That won't work and it worked before just because golded
was not initializing correctly!

Ok, so how do I fix it? ;)

Yesterday I did some experimenting with the latest golded (commit: "call setlocale() before initscr() (#86)"), trying different ways to start golded and different golded xlat configurations. I found 2 setups that work for me, so the messages with pseudo graphics in the STATS area look ok:

My golded.cfg for both:

include charsets.cfg

XLATIMPORT CP437
XLATLOCALSET CP850
XLATEXPORT CP850


And starting golded in a local Konsole terminal "TERM=xterm" with the default charset set to utf-8:

# sudo -u fido LANG=en_EN.CP850 luit -encoding 'CP850' /usr/local/bin/golded -f


And in the same kind of terminal but with the default charset set to cp850, this one works:

# sudo -u fido LANG=en_US.CP850 /usr/local/bin/golded -f


I will still have to test these in a putty terminal later during the coming week.

And messages which have utf-8 characters, like the ones from Michiel in the UTF area, don't display properly (of course).
I did try adding:

XLATCHARSET UTF-8 CP850 utf8_850.chs

But that didn't help. Of course only a small subset of utf-8 characters can be displayed with cp850...


Bye, Wilfred.

--- FMail-lnx64 2.2.1.1
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

09 Mar 24 16:57, you wrote to me:

OK. Your terminal works in IBM850, but you run golded may try to
run in UTF-8. That won't work and it worked before just because
golded was not initializing correctly!

Ok, so how do I fix it? ;)

Yesterday I did some experimenting with the latest golded (commit:
"call setlocale() before initscr() (#86)"), trying different ways to start golded and different golded xlat configurations. I found 2
setups that work for me, so the messages with pseudo graphics in the STATS area look ok:

My golded.cfg for both:

include charsets.cfg

XLATIMPORT CP437
XLATLOCALSET CP850
XLATEXPORT CP850

Do not use xlatexport cp850. This parameter is used to select charset which will be used when you write messages to message base. Unless you really want cp850. I assume you want to have cp437 there.

XlatImport on other side will use cp437 if message has no CHRS kludge.

And starting golded in a local Konsole terminal "TERM=xterm" with the default charset set to utf-8:

# sudo -u fido LANG=en_EN.CP850 luit -encoding 'CP850' /usr/local/bin/golded -f

If you use luit - than having xterm in UTF-8 is correct.

And in the same kind of terminal but with the default charset set to cp850, this one works:

# sudo -u fido LANG=en_US.CP850 /usr/local/bin/golded -f

This is correct commandline to if you have your terminal in cp850. Just choose what works better for you. Probably luit and terminal in UTF-8 is most convenient.

I will still have to test these in a putty terminal later during the coming week.

Putty will work fine. Only I suggest you to change terminal type in putty from xterm (they use if by default) to putty or putty-256color. It's in Connection->Data section.

And messages which have utf-8 characters, like the ones from Michiel
in the UTF area, don't display properly (of course). I did try adding:

XLATCHARSET UTF-8 CP850 utf8_850.chs

But that didn't help. Of course only a small subset of utf-8
characters can be displayed with cp850...

Golded cannot read from UTF-8. There is no Unicode support. You must use another editor or GoldEd with external editor to work with Unicode.

It may convert from one-byte charsets to UTF-8, but not vise versa.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-09 12:49:34, you wrote to me:

Yesterday I did some experimenting with the latest golded (commit:
"call setlocale() before initscr() (#86)"), trying different ways to
start golded and different golded xlat configurations. I found 2
setups that work for me, so the messages with pseudo graphics in the
STATS area look ok:

My golded.cfg for both:

include charsets.cfg

XLATIMPORT CP437
XLATLOCALSET CP850
XLATEXPORT CP850

Do not use xlatexport cp850. This parameter is used to select charset which
will be used when you write messages to message base. Unless you really want cp850. I assume you want to have cp437 there.

My terminal uses cp850, so it's what I write my messages in. If I translate it to cp437, it will possibly loose some characters that are not supported in both. So I think I want it. ;-)

And starting golded in a local Konsole terminal "TERM=xterm" with
the default charset set to utf-8:

# sudo -u fido LANG=en_EN.CP850 luit -encoding 'CP850'
/usr/local/bin/golded -f

If you use luit - than having xterm in UTF-8 is correct.

And in the same kind of terminal but with the default charset set to
cp850, this one works:

# sudo -u fido LANG=en_US.CP850 /usr/local/bin/golded -f

This is correct commandline to if you have your terminal in cp850. Just choose what works better for you. Probably luit and terminal in UTF-8 is most convenient.

I'll see after I have tested with putty...

I will still have to test these in a putty terminal later during the
coming week.

Putty will work fine. Only I suggest you to change terminal type in putty from xterm (they use if by default)

I think it's set to 'linux' in putty.

to putty or putty-256color. It's in Connection->Data section.

Ok, I'll try that.

And messages which have utf-8 characters, like the ones from Michiel
in the UTF area, don't display properly (of course). I did try
adding:

XLATCHARSET UTF-8 CP850 utf8_850.chs

But that didn't help. Of course only a small subset of utf-8
characters can be displayed with cp850...

Golded cannot read from UTF-8. There is no Unicode support. You must use another editor or GoldEd with external editor to work with Unicode.

It may convert from one-byte charsets to UTF-8, but not vise versa.

Isn't this for incoming messages in utf-8, to show them in your local charset, as far as possible?


Bye, Wilfred.

--- FMail-lnx64 2.2.1.1
* Origin: FMail development HQ (2:280/464)

On Sat, 9 Mar 2024 18:49:34 -0700, Vitaliy Aksyonov -> Wilfred Van
Velzen wrote:

Golded cannot read from UTF-8. There is no Unicode support. You must
use another editor or GoldEd with external editor to work with
Unicode.

It may convert from one-byte charsets to UTF-8, but not vise versa.

We know it is unsupported, but I'm pretty sure I've said multiple times
it can display in Golded just fine, with CHRS kludge, UTF-8 Cyrillic
characters (MANY other characters display fine, also):

https://pharcyde.org/golded-utf8.png

It indeed can be done. The problem is everyone's terminals and what else is being used to run golded is different in every case.

Regards,
Nick

... "Take my advice, I don't use it anyway."
--- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:115.0) Gecko/20100101 Thunderb
* Origin: _thePharcyde distribution system (Wisconsin) (1:154/10)

Hi Vitaliy,

On 2024-03-09 21:53:51, I wrote to you:

And starting golded in a local Konsole terminal "TERM=xterm" with
the default charset set to utf-8:

# sudo -u fido LANG=en_EN.CP850 luit -encoding 'CP850'
/usr/local/bin/golded -f

If you use luit - than having xterm in UTF-8 is correct.

This one also works with putty (which is set to use utf-8 as charset).

And in the same kind of terminal but with the default charset set
to cp850, this one works:

# sudo -u fido LANG=en_US.CP850 /usr/local/bin/golded -f

This is correct commandline to if you have your terminal in cp850.
Just choose what works better for you. Probably luit and terminal in
UTF-8 is most convenient.

I'll see after I have tested with putty...

I'll probably switch to this setup since it also works with putty.

I will still have to test these in a putty terminal later during
the coming week.

Putty will work fine. Only I suggest you to change terminal type in
putty from xterm (they use if by default)

I think it's set to 'linux' in putty.

Switching from 'linux' to 'putty' makes things worse:

https://paste.opensuse.org/pastes/8a290fabf761

Also some keys, like the F3 key stopped working. So I keep my terminal type as 'linux'.

to putty or putty-256color. It's in Connection->Data section.

'putty-256color' makes no difference.


Bye, Wilfred.

--- FMail-lnx64 2.2.1.1
* Origin: FMail development HQ (2:280/464)

Hello Nicholas.

10 Mar 24 14:56, you wrote to me:

Golded cannot read from UTF-8. There is no Unicode support. You
must use another editor or GoldEd with external editor to work
with Unicode.

It may convert from one-byte charsets to UTF-8, but not vise
versa.

We know it is unsupported, but I'm pretty sure I've said multiple
times it can display in Golded just fine, with CHRS kludge, UTF-8
Cyrillic characters (MANY other characters display fine, also):

https://pharcyde.org/golded-utf8.png

We're trying to find, what could be wrong in GoldEd using ncurses. And it reproduces, so probably we'll find some solution.

It indeed can be done. The problem is everyone's terminals and what
else is being used to run golded is different in every case.

I know. But there are many other problems, not just displaying unicode symbols, but also splitting long paragraphs of non-english letters. They may be 2-4 bytes long in UTF-8 and It will be weird in reader window.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hello Wilfred.

11 Mar 24 09:26, you wrote to me:

And starting golded in a local Konsole terminal "TERM=xterm"
with the default charset set to utf-8:

# sudo -u fido LANG=en_EN.CP850 luit -encoding 'CP850'
/usr/local/bin/golded -f

If you use luit - than having xterm in UTF-8 is correct.

This one also works with putty (which is set to use utf-8 as charset).

Yep. I've tried luit in my configuration and it works totally fine.

And in the same kind of terminal but with the default charset
set to cp850, this one works:

# sudo -u fido LANG=en_US.CP850 /usr/local/bin/golded -f

This is correct commandline to if you have your terminal in
cp850. Just choose what works better for you. Probably luit and
terminal in UTF-8 is most convenient.

I'll see after I have tested with putty...

I'll probably switch to this setup since it also works with putty.

It's definitely more convenient.

I will still have to test these in a putty terminal later
during the coming week.

Putty will work fine. Only I suggest you to change terminal type
in putty from xterm (they use if by default)

I think it's set to 'linux' in putty.

Switching from 'linux' to 'putty' makes things worse:

https://paste.opensuse.org/pastes/8a290fabf761

That may be because terminfo for putty doesn't exist on your system. You may need to install additional packages. I have putty-256color and it works perfect.

How to check.
Run infocmp -D
This will show you which directories are used to find terminfo.
Then search in those directories, is putty available. If not - you may need install package like ncurses-term. May have different name on your system.

Let me know it that helped please.

Also some keys, like the F3 key stopped working. So I keep my
terminal
type as 'linux'.

to putty or putty-256color. It's in Connection->Data section.

'putty-256color' makes no difference.

putty-256color just add more colors support. F keys are broken most probably because terminfo is missing.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-12 19:34:48, you wrote to me:

This is correct commandline to if you have your terminal in
cp850. Just choose what works better for you. Probably luit and
terminal in UTF-8 is most convenient.

I'll see after I have tested with putty...

I'll probably switch to this setup since it also works with putty.

It's definitely more convenient.

I changed to using luit, for both the putty and konsole terminals for golded.

I do now have the problem that sometimes (not most of the time), the terminal is in strange state after closing golded. I've fixed that by adding the 'reset' command as the last command in the script that starts golded.

Putty will work fine. Only I suggest you to change terminal type
in putty from xterm (they use if by default)

I think it's set to 'linux' in putty.

Switching from 'linux' to 'putty' makes things worse:

https://paste.opensuse.org/pastes/8a290fabf761

That may be because terminfo for putty doesn't exist on your system. You may need to install additional packages. I have putty-256color and it works
perfect.

How to check.
Run infocmp -D
This will show you which directories are used to find terminfo.
Then search in those directories, is putty available. If not - you may need
install package like ncurses-term. May have different name on your system.

I had already checked this:

# ls -l /etc/termcap
lrwxrwxrwx 1 root root 23 2015-10-29 21:23:18 termcap -> /usr/share/misc/termcap

# grep putty /etc/termcap
putty|PuTTY terminal emulator:\
vt100-putty|Reset PuTTY to pure vt100:\
putty-256color|PuTTY 0.58 with xterm 256-colors:\
putty-vt100|VT100+ keyboard layout:\
putty-sco|putty with SCO function keys:\


infocmp shows more or less the same:

# infocmp -D
/usr/share/terminfo

# find /usr/share/terminfo -iname 'putty*'
/usr/share/terminfo/p/putty
/usr/share/terminfo/p/putty-256color
/usr/share/terminfo/p/putty-sco
/usr/share/terminfo/p/putty-vt100

Let me know it that helped please.

So terminfo for putty already exists on my system...

Also some keys, like the F3 key stopped working. So I keep my
terminal type as 'linux'.

to putty or putty-256color. It's in Connection->Data section.

'putty-256color' makes no difference.

putty-256color just add more colors support. F keys are broken most probably because terminfo is missing.

That is not the case...


Bye, Wilfred.

--- FMail-lnx64 2.2.1.1
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

13 Mar 24 08:45, you wrote to me:

This is correct commandline to if you have your terminal in
cp850. Just choose what works better for you. Probably luit
and terminal in UTF-8 is most convenient.

I'll see after I have tested with putty...

I'll probably switch to this setup since it also works with
putty.

It's definitely more convenient.

I changed to using luit, for both the putty and konsole terminals for golded.

I do now have the problem that sometimes (not most of the time), the terminal is in strange state after closing golded. I've fixed that by adding the 'reset' command as the last command in the script that
starts golded.

It could be something wrong with ncurses deinit. I've never saw that. But I usually doesn't use session for anything else than GoldEd. And usually just log off.

Putty will work fine. Only I suggest you to change terminal
type in putty from xterm (they use if by default)

I think it's set to 'linux' in putty.

Switching from 'linux' to 'putty' makes things worse:

https://paste.opensuse.org/pastes/8a290fabf761

That may be because terminfo for putty doesn't exist on your
system. You may need to install additional packages. I have
putty-256color and it works perfect.

How to check.
Run infocmp -D
This will show you which directories are used to find terminfo.
Then search in those directories, is putty available. If not -
you may need install package like ncurses-term. May have
different name on your system.

I had already checked this:

# ls -l /etc/termcap
lrwxrwxrwx 1 root root 23 2015-10-29 21:23:18 termcap -> /usr/share/misc/termcap

# grep putty /etc/termcap
putty|PuTTY terminal emulator:\
vt100-putty|Reset PuTTY to pure vt100:\
putty-256color|PuTTY 0.58 with xterm 256-colors:\
putty-vt100|VT100+ keyboard layout:\
putty-sco|putty with SCO function keys:\

infocmp shows more or less the same:

# infocmp -D
/usr/share/terminfo

# find /usr/share/terminfo -iname 'putty*'
/usr/share/terminfo/p/putty
/usr/share/terminfo/p/putty-256color
/usr/share/terminfo/p/putty-sco
/usr/share/terminfo/p/putty-vt100

Let me know it that helped please.

So terminfo for putty already exists on my system...

Also some keys, like the F3 key stopped working. So I keep my
terminal type as 'linux'.

to putty or putty-256color. It's in Connection->Data section.

'putty-256color' makes no difference.

putty-256color just add more colors support. F keys are broken
most probably because terminfo is missing.

That is not the case...

I spend a lot of time configuring my Putty to work correctly. I'd suggest you first make it working without luit. You may need to play with Putty's settings. And TERM=putty works the best for me. What I mean - it may work and it's just configuration thing. It should be totally fine for you too because you have correct termnifo.


Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-13 14:19:04, you wrote to me:

putty-256color just add more colors support. F keys are broken
most probably because terminfo is missing.

That is not the case...

I spend a lot of time configuring my Putty to work correctly. I'd suggest you first make it working without luit.

Then I have to set my terminals to cp850 charset. I rather keep them as utf8

You may need to play with Putty's settings. And TERM=putty works the
best for me. What I mean - it may work and it's just configuration
thing. It should be totally fine for you too because you have correct termnifo.

I can spend a lot of time on it, or keep my current working config... I see no gain, spending the time.

Bye, Wilfred.

--- FMail-lnx64 2.2.1.1
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

13 Mar 24 21:58, you wrote to me:

putty-256color just add more colors support. F keys are broken
most probably because terminfo is missing.

That is not the case...

I spend a lot of time configuring my Putty to work correctly. I'd
suggest you first make it working without luit.

Then I have to set my terminals to cp850 charset. I rather keep them
as utf8

Only temporary. F keys shall not be dependent on locale.

You may need to play with Putty's settings. And TERM=putty works
the best for me. What I mean - it may work and it's just
configuration thing. It should be totally fine for you too
because you have correct termnifo.

I can spend a lot of time on it, or keep my current working config...
I see no gain, spending the time.

Sure. If that works for you. I'm just trying to help. :)

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-13 15:31:20, you wrote to me:

I spend a lot of time configuring my Putty to work correctly. I'd
suggest you first make it working without luit.

Then I have to set my terminals to cp850 charset. I rather keep them
as utf8

Only temporary. F keys shall not be dependent on locale.

I suspect it might have to do with different escape sequences, or handling of, for the function keys in the 'linux' and 'putty' settings.

You may need to play with Putty's settings. And TERM=putty works
the best for me. What I mean - it may work and it's just
configuration thing. It should be totally fine for you too
because you have correct termnifo.

I can spend a lot of time on it, or keep my current working config...
I see no gain, spending the time.

Sure. If that works for you. I'm just trying to help. :)

And that is highly appreciated!

Bye, Wilfred.

--- FMail-lnx64 2.2.1.1
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

14 Mar 24 08:55, you wrote to me:

I spend a lot of time configuring my Putty to work correctly.
I'd suggest you first make it working without luit.

Then I have to set my terminals to cp850 charset. I rather keep
them as utf8

Only temporary. F keys shall not be dependent on locale.

I suspect it might have to do with different escape sequences, or handling of, for the function keys in the 'linux' and 'putty'
settings.

You're completely right. That's the reason. And when you change terminal type, ncurses expects different escape sequences.

My settings are:
terminal type putty

Then in Terminal->Keyboard:

The Backspace key -> Control-?
The Home and End keys -> Standard
The Function keys and keypad -> ESC[n~
Shift/Ctrl/Alt with the arrow keys -> Ctrl toggles app mode
Initial state of cursor keys -> Normal
Initial state of numeric keypad -> Normal
AltGf acts as Compose key -> off
Control-Alt is different from AltGf -> on

In Terminal->Features all is off except "Disable application keypad mode"

You may need to play with Putty's settings. And TERM=putty
works the best for me. What I mean - it may work and it's just
configuration thing. It should be totally fine for you too
because you have correct termnifo.

I can spend a lot of time on it, or keep my current working
config... I see no gain, spending the time.

Sure. If that works for you. I'm just trying to help. :)

And that is highly appreciated!

Try my settings. It could work.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-14 07:19:32, you wrote to me:

I suspect it might have to do with different escape sequences, or
handling of, for the function keys in the 'linux' and 'putty'
settings.

You're completely right. That's the reason. And when you change terminal type, ncurses expects different escape sequences.

My settings are:
terminal type putty

Then in Terminal->Keyboard:

The Backspace key -> Control-?
The Home and End keys -> Standard
The Function keys and keypad -> ESC[n~

That was the one that was different for me.

I also had to Disable remote-controled terminal resizing, otherwise my terminal would resize to 80 chars wide after closing golded.

Shift/Ctrl/Alt with the arrow keys -> Ctrl toggles app mode
Initial state of cursor keys -> Normal
Initial state of numeric keypad -> Normal
AltGf acts as Compose key -> off
Control-Alt is different from AltGf -> on

In Terminal->Features all is off except "Disable application keypad mode"

Try my settings. It could work.

It fixed the keyboard problems. But not how golded looks. The line drawing characters are still the 'qqqqqq' kind.
It doesn't matter if I use luit or not.

What did help, was enabling in Window/Tranlation: Enable VT100 line drawing even in UTF-8 mode
So now I can use the 'putty-256color' terminal type. :-)


Bye, Wilfred.

--- FMail-lnx64 2.2.1.1
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

14 Mar 24 15:25, you wrote to me:

I suspect it might have to do with different escape sequences,
or handling of, for the function keys in the 'linux' and
'putty' settings.

You're completely right. That's the reason. And when you change
terminal type, ncurses expects different escape sequences.

My settings are:
terminal type putty

Then in Terminal->Keyboard:

The Backspace key -> Control-?
The Home and End keys -> Standard
The Function keys and keypad -> ESC[n~

That was the one that was different for me.

I also had to Disable remote-controled terminal resizing, otherwise my terminal would resize to 80 chars wide after closing golded.

Good.

Shift/Ctrl/Alt with the arrow keys -> Ctrl toggles app mode
Initial state of cursor keys -> Normal
Initial state of numeric keypad -> Normal
AltGf acts as Compose key -> off
Control-Alt is different from AltGf -> on

In Terminal->Features all is off except "Disable application
keypad mode"

Try my settings. It could work.

It fixed the keyboard problems. But not how golded looks. The line drawing characters are still the 'qqqqqq' kind. It doesn't matter if I use luit or not.

What did help, was enabling in Window/Tranlation: Enable VT100 line drawing even in UTF-8 mode So now I can use the 'putty-256color'
terminal type. :-)

The only difference that I don't run golded in UTF, that's why you had to change this parameter. Glad it works for you!

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-14 10:32:22, you wrote to me:

What did help, was enabling in Window/Tranlation: Enable VT100 line
drawing even in UTF-8 mode So now I can use the 'putty-256color'
terminal type. :-)

The only difference that I don't run golded in UTF, that's why you had to change this parameter. Glad it works for you!

My terminal is UTF, bug I start golded with 'LANG=en_EN.CP850 luit -encoding 'CP850'', so I don't run golded in UTF either. Or do you mean something else?

Bye, Wilfred.

--- FMail-lnx64 2.2.1.1
* Origin: FMail development HQ (2:280/464)

Hello, Wilfred!

Thursday March 14 2024 17:37, from Wilfred van Velzen -> Vitaliy Aksyonov:

My terminal is UTF, bug I start golded with 'LANG=en_EN.CP850 luit -encoding 'CP850'', so I don't run golded in UTF either. Or do you
mean something else?

Make sure your luit has that encoding.
$ luit -list | grep CP850

Mine doesn't
$ luit -list | grep -ic CP850
0

Best Regards, Nil
--- GoldED+/LNX 1.1.5
* Origin: Linux 2.6.32-042stab145.3 (2:5015/46)

Hi Nil,

On 2024-03-14 20:03:22, you wrote to me:

My terminal is UTF, bug I start golded with 'LANG=en_EN.CP850 luit
-encoding 'CP850'', so I don't run golded in UTF either. Or do you
mean something else?

Make sure your luit has that encoding.
$ luit -list | grep CP850

# luit -list | grep CP850
CP850: GL -> G0, GR -> G2, G0: ASCII, G2: CP 850

And otherwise it probably wouldn't have worked. ;-)

But thanks for the tip!


Bye, Wilfred.

--- FMail-lnx64 2.2.1.1
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

14 Mar 24 17:37, you wrote to me:

What did help, was enabling in Window/Tranlation: Enable VT100
line drawing even in UTF-8 mode So now I can use the
'putty-256color' terminal type. :-)

The only difference that I don't run golded in UTF, that's why
you had to change this parameter. Glad it works for you!

My terminal is UTF, bug I start golded with 'LANG=en_EN.CP850 luit -encoding 'CP850'', so I don't run golded in UTF either. Or do you
mean something else?

Yep. That is what I meant. Anyway I glad it works for you now.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-14 12:29:14, you wrote to me:

What did help, was enabling in Window/Tranlation: Enable VT100
line drawing even in UTF-8 mode So now I can use the
'putty-256color' terminal type. :-)

The only difference that I don't run golded in UTF, that's why
you had to change this parameter. Glad it works for you!

My terminal is UTF, bug I start golded with 'LANG=en_EN.CP850 luit
-encoding 'CP850'', so I don't run golded in UTF either. Or do you
mean something else?

Yep. That is what I meant. Anyway I glad it works for you now.

I'm still seeing an issue with some messages in German areas. For instance:

https://paste.opensuse.org/pastes/0cc6a7ecd695

Some characters are displayed as '~A', where they should be able to be displayed in their right form, while others are displayed correctly.

The two High ascii characters in the marked area are:

\x81 ; latin small letter u with diaeresis
\xE1 ; greek small letter beta (here used as german ss)

The last character in the line, an 'e', isn't even displayed, probably because of the expansion of 1 of the characters to 2 characters.

These characters are the same in CP437 and CP850. So my CP850 terminal should display them correctly.
It seems to come from Golded itself, or maybe the ncurses library, because it doesn't matter if I use luit for starting Golded or not, or running it in my remote putty terminal, or my local konsole terminal. The \x81 character is always displayed as ~A ... Strange!?


Bye, Wilfred.

--- FMail-lnx64 2.3.0.1-B20240319
* Origin: FMail development HQ (2:280/464)

Hi Vitaliy,

On 2024-03-22 09:38:52, I wrote to you:

I'm still seeing an issue with some messages in German areas. For instance:

https://paste.opensuse.org/pastes/0cc6a7ecd695

Some characters are displayed as '~A', where they should be able to be displayed in their right form, while others are displayed correctly.

The two High ascii characters in the marked area are:

\x81 ; latin small letter u with diaeresis
\xE1 ; greek small letter beta (here used as german ss)

The last character in the line, an 'e', isn't even displayed, probably because
of the expansion of 1 of the characters to 2 characters.

These characters are the same in CP437 and CP850. So my CP850 terminal should
display them correctly. It seems to come from Golded itself, or maybe the ncurses library, because it doesn't matter if I use luit for starting Golded
or not, or running it in my remote putty terminal, or my local konsole terminal. The \x81 character is always displayed as ~A ... Strange!?

Btw: My terminal seems fine with displaying the CP850 high ascii characters (despite the warning):

https://paste.opensuse.org/pastes/8bcb9d2ecfdf


Bye, Wilfred.

--- FMail-lnx64 2.3.0.1-B20240319
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

22 Mar 24 10:45, you wrote to me:

I'm still seeing an issue with some messages in German areas.
For
instance:

https://paste.opensuse.org/pastes/0cc6a7ecd695

Some characters are displayed as '~A', where they should be able
to be displayed in their right form, while others are displayed
correctly.

The two High ascii characters in the marked area are:

\x81 ; latin small letter u with diaeresis
\xE1 ; greek small letter beta (here used as german ss)

The last character in the line, an 'e', isn't even displayed,
probably because of the expansion of 1 of the characters to 2
characters.

These characters are the same in CP437 and CP850. So my CP850
terminal should display them correctly. It seems to come from
Golded itself, or maybe the ncurses library, because it doesn't
matter if I use luit for starting Golded or not, or running it
in my remote putty terminal, or my local konsole terminal. The
\x81 character is always displayed as ~A ... Strange!?

Btw: My terminal seems fine with displaying the CP850 high ascii characters (despite the warning):

https://paste.opensuse.org/pastes/8bcb9d2ecfdf

Looks like your luit doesn't support CP850 or you don't have en_US.CP850. There are some encodings which luit list, but doesn't support actually. For example, mine lists CP866, but doesn't work with it.

Does it present in `locale -a` output?

Have you tried to run that script without luit? You don't even need to change locale for it - just encoding in terminal.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-22 07:04:30, you wrote to me:

Btw: My terminal seems fine with displaying the CP850 high ascii
characters (despite the warning):

https://paste.opensuse.org/pastes/8bcb9d2ecfdf

Looks like your luit doesn't support CP850 or you don't have en_US.CP850. There are some encodings which luit list, but doesn't support actually. For
example, mine lists CP866, but doesn't work with it.

Does it present in `locale -a` output?

I don't know if that says much, because mostly there are just the xx_XX and xx_XX.utf8 versions of the encodings. To give you a sample:

# locale -a | grep en_
en_AG
en_AU
en_AU.utf8
en_BE
en_BE.utf8
en_BE@euro
en_BW
en_BW.utf8
en_CA
en_CA.utf8
en_DK
en_DK.utf8
en_GB
en_GB.iso885915
en_GB.utf8
en_HK
en_HK.utf8
en_IE
en_IE.utf8
en_IE@euro
en_IN
en_NG
en_NZ
en_NZ.utf8
en_PH
en_PH.utf8
en_SG
en_SG.utf8
en_US
en_US.iso885915
en_US.utf8
en_ZA
en_ZA.utf8
en_ZM
en_ZW
en_ZW.utf8

Does this mean there is just an utf8 charset and an unspecified one for almost every language-country? That doesn't seem logical!

Doesn't this show you what encodings luit supports:

# luit -list
Known locale encodings:

C: GL -> G0, GR -> G2, G0: ASCII, G2: ISO 8859-1
POSIX: GL -> G0, GR -> G2, G0: ASCII, G2: ISO 8859-1
US-ASCII: GL -> G0, GR -> G2, G0: ASCII, G2: ISO 8859-1
...
CP850: GL -> G0, GR -> G2, G0: ASCII, G2: CP 850
...

Known charsets (not all may be available):

ISO 646 (1973) (ISO 2022, 94 codes)
ASCII (ISO 2022, 94 codes)
...
CP 437 (128 codes)
CP 850 (128 codes)
CP 852 (128 codes)
...

So luit seems to know about CP850...

Have you tried to run that script without luit? You don't even need to change locale for it - just encoding in terminal.

https://paste.opensuse.org/pastes/574e349fadaf

So without luit it doesn't display anything useful. With luit, although you get the warning, it does display the right characters for CP850 !?


Bye, Wilfred.

--- FMail-lnx64 2.3.0.1-B20240319
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

23 Mar 24 13:57, you wrote to me:

Btw: My terminal seems fine with displaying the CP850 high
ascii characters (despite the warning):

https://paste.opensuse.org/pastes/8bcb9d2ecfdf

Looks like your luit doesn't support CP850 or you don't have
en_US.CP850. There are some encodings which luit list, but
doesn't support actually. For example, mine lists CP866, but
doesn't work with it.

Does it present in `locale -a` output?

I don't know if that says much, because mostly there are just the
xx_XX and xx_XX.utf8 versions of the encodings. To give you a sample:

# locale -a | grep en_

[...skipped...]

Does this mean there is just an utf8 charset and an unspecified one
for almost every language-country? That doesn't seem logical!

It just mean that your system doesn't have necessary locale installed. And that perfectly explain why luit shows all chars, but GoldEd doesn't.

Luit converts them using internal tables to UTF, but GoldEd tries to use en_EN.CP850, which is missing. That's why it doesn't understand that letters with codes > 127 are letters.

You need to install or generate this locale and GoldEd will show those letters! What Linux distribution do you use? Do you need help with locale generation?

Doesn't this show you what encodings luit supports:

# luit -list
Known locale encodings:

C: GL -> G0, GR -> G2, G0: ASCII, G2: ISO 8859-1
POSIX: GL -> G0, GR -> G2, G0: ASCII, G2: ISO 8859-1
US-ASCII: GL -> G0, GR -> G2, G0: ASCII, G2: ISO 8859-1
...
CP850: GL -> G0, GR -> G2, G0: ASCII, G2: CP 850
...

Known charsets (not all may be available):

ISO 646 (1973) (ISO 2022, 94 codes)
ASCII (ISO 2022, 94 codes)
...
CP 437 (128 codes)
CP 850 (128 codes)
CP 852 (128 codes)
...

So luit seems to know about CP850...

Yes. And that's good!

Have you tried to run that script without luit? You don't even
need to change locale for it - just encoding in terminal.

https://paste.opensuse.org/pastes/574e349fadaf

So without luit it doesn't display anything useful. With luit,
although you get the warning, it does display the right characters for CP850 !?

Yep. I meant to run that script without luit in terminal configured for CP850. But you don't need to do that anymore as long as we found root cause already.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-24 11:12:14, you wrote to me:

You need to install or generate this locale and GoldEd will show those letters! What Linux distribution do you use?

wilnux5:/etc # cat os-release
NAME="openSUSE Leap"
VERSION="42.1"
VERSION_ID="42.1"
PRETTY_NAME="openSUSE Leap 42.1 (x86_64)"
ID=opensuse
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:opensuse:42.1" BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://opensuse.org/"
ID_LIKE="suse"

(Yes, it's old ;-))

Do you need help with locale generation?

I think I found out how to do this on my system.

First I used my systems package manager to install: "glibc-i18ndata - Database Sources for 'locale'"

Afterwards this command ran without any output:

# localedef --no-archive -f IBM850 -i en_US en_US.CP850
#

And this directory was created with contents:

/usr/lib/locale/en_US.cp850

And 'locale -a -v' now shows:
...
locale: en_US directory: /usr/lib/locale/en_US -------------------------------------------------------------------------------
title | English locale for the USA
source | Free Software Foundation, Inc.
address | http://www.gnu.org/software/libc/
email | bug-glibc-locales@gnu.org
language | English
territory | USA
revision | 1.0
date | 2000-06-24
codeset | ISO-8859-1

locale: en_US.cp850 directory: /usr/lib/locale/en_US.cp850 -------------------------------------------------------------------------------
title | English locale for the USA
source | Free Software Foundation, Inc.
address | http://www.gnu.org/software/libc/
email | bug-glibc-locales@gnu.org
language | English
territory | USA
revision | 1.0
date | 2000-06-24
codeset | IBM850
...

But golded output is borked now, in my current utf-8 configured putty terminal:

https://paste.opensuse.org/pastes/2a5c7f2fbdc4

It doesn't matter if I use luit or not, they are displayed the same.
Also the ~A characters for messages with CHRS: CP437 in the german areas are still there.

And this is still the same:

https://paste.opensuse.org/pastes/f3961b7ea085


Bye, Wilfred.

--- FMail-lnx64 2.3.0.1-B20240319
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

25 Mar 24 09:01, you wrote to me:

You need to install or generate this locale and GoldEd will show
those letters! What Linux distribution do you use?

wilnux5:/etc # cat os-release
NAME="openSUSE Leap"
VERSION="42.1"
VERSION_ID="42.1"
PRETTY_NAME="openSUSE Leap 42.1 (x86_64)"
ID=opensuse
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:opensuse:42.1" BUG_REPORT_URL="https://bugs.opensuse.org" HOME_URL="https://opensuse.org/"
ID_LIKE="suse"

(Yes, it's old ;-))

Got it. Not too old for FidoNet. :)

Do you need help with locale generation?

I think I found out how to do this on my system.

First I used my systems package manager to install: "glibc-i18ndata - Database Sources for 'locale'"

Afterwards this command ran without any output:

# localedef --no-archive -f IBM850 -i en_US en_US.CP850
#

And this directory was created with contents:

/usr/lib/locale/en_US.cp850

And 'locale -a -v' now shows:
...
locale: en_US directory: /usr/lib/locale/en_US ---------------------------------------------------------------------- ---------
title | English locale for the USA
source | Free Software Foundation, Inc.
address | http://www.gnu.org/software/libc/
email | bug-glibc-locales@gnu.org
language | English
territory | USA
revision | 1.0
date | 2000-06-24
codeset | ISO-8859-1

locale: en_US.cp850 directory: /usr/lib/locale/en_US.cp850 ---------------------------------------------------------------------- ---------
title | English locale for the USA
source | Free Software Foundation, Inc.
address | http://www.gnu.org/software/libc/
email | bug-glibc-locales@gnu.org
language | English
territory | USA
revision | 1.0
date | 2000-06-24
codeset | IBM850
...

This is what you need. Good.

But golded output is borked now, in my current utf-8 configured putty terminal:

https://paste.opensuse.org/pastes/2a5c7f2fbdc4

This is exactly how I saw it on my computer, when was using pseudo-graphics with wrong or missing locale.

It doesn't matter if I use luit or not, they are displayed the same.
Also the ~A characters for messages with CHRS: CP437 in the german
areas are still there.

And this is still the same:

https://paste.opensuse.org/pastes/f3961b7ea085

This looks like locale is not used. I saw such pictures when Strange.

Do you use latest GoldEd build? Do you still run it with LANG=en_US.cp850? In some message I saw en_EN.CP850.

Could you also run:
LANG=en_US.cp850 locale

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-25 07:14:50, you wrote to me:

PRETTY_NAME="openSUSE Leap 42.1 (x86_64)"

(Yes, it's old ;-))

Got it. Not too old for FidoNet. :)

Nope. ;-)

locale: en_US.cp850 directory: /usr/lib/locale/en_US.cp850

This is what you need. Good.

But golded output is borked now, in my current utf-8 configured putty
terminal:

https://paste.opensuse.org/pastes/2a5c7f2fbdc4

This is exactly how I saw it on my computer, when was using pseudo-graphics
with wrong or missing locale.

The locale is there, so is it wrong?

It doesn't matter if I use luit or not, they are displayed the same.
Also the ~A characters for messages with CHRS: CP437 in the german
areas are still there.

And this is still the same:

https://paste.opensuse.org/pastes/f3961b7ea085

This looks like locale is not used. I saw such pictures when Strange.

Do you use latest GoldEd build?

Almost. I'm using this one:

commit 4b6c754756d0fa96c0c3210d6ed0b63d49ec8e6a
Author: Vitaliy Aksyonov <18148062+vitaliy-aksyonov@users.noreply.github.com> Date: Wed Mar 6 13:38:52 2024 -0700

call setlocale() before initscr() (#86)

See section Initialization in man 3 ncurses.
https://www.man7.org/linux/man-pages/man3/ncurses.3x.html
Locale shall be initialized before ncurses initialization.

The latest one doesn't seem so change anything regarding screen output.

Do you still run it with LANG=en_US.cp850?

Yes.

In some message I saw en_EN.CP850.

The localdef command, created the directory with lowercase 'cp850' although I specified it with uppercase 'CP850'. It also shows it with lowercase 'cp' when locale -a is executed. So I switched to specifying it as lowercase in my golded start script. But case probably doesn't matter.

Could you also run:
LANG=en_US.cp850 locale

Here are some tries:

wilnux5:/home/fido/log # locale
LANG=POSIX
LC_CTYPE=en_US.UTF-8
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

wilnux5:/home/fido/log # LANG=en_US.cp850 locale
LANG=en_US.cp850
LC_CTYPE=en_US.UTF-8
LC_NUMERIC="en_US.cp850"
LC_TIME="en_US.cp850"
LC_COLLATE="en_US.cp850"
LC_MONETARY="en_US.cp850"
LC_MESSAGES="en_US.cp850"
LC_PAPER="en_US.cp850"
LC_NAME="en_US.cp850"
LC_ADDRESS="en_US.cp850"
LC_TELEPHONE="en_US.cp850"
LC_MEASUREMENT="en_US.cp850"
LC_IDENTIFICATION="en_US.cp850"
LC_ALL=

wilnux5:/home/fido/log # LANG=en_US.CP850 locale
LANG=en_US.CP850
LC_CTYPE=en_US.UTF-8
LC_NUMERIC="en_US.CP850"
LC_TIME="en_US.CP850"
LC_COLLATE="en_US.CP850"
LC_MONETARY="en_US.CP850"
LC_MESSAGES="en_US.CP850"
LC_PAPER="en_US.CP850"
LC_NAME="en_US.CP850"
LC_ADDRESS="en_US.CP850"
LC_TELEPHONE="en_US.CP850"
LC_MEASUREMENT="en_US.CP850"
LC_IDENTIFICATION="en_US.CP850"
LC_ALL=

wilnux5:/home/fido/log # sudo -u fido LANG=en_US.cp850 locale
LANG=en_US.cp850
LC_CTYPE=en_US.UTF-8
LC_NUMERIC="en_US.cp850"
LC_TIME="en_US.cp850"
LC_COLLATE="en_US.cp850"
LC_MONETARY="en_US.cp850"
LC_MESSAGES="en_US.cp850"
LC_PAPER="en_US.cp850"
LC_NAME="en_US.cp850"
LC_ADDRESS="en_US.cp850"
LC_TELEPHONE="en_US.cp850"
LC_MEASUREMENT="en_US.cp850"
LC_IDENTIFICATION="en_US.cp850"
LC_ALL=


Bye, Wilfred.

--- FMail-lnx64 2.3.0.1-B20240319
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

25 Mar 24 15:13, you wrote to me:

locale: en_US.cp850 directory: /usr/lib/locale/en_US.cp850

This is what you need. Good.

But golded output is borked now, in my current utf-8 configured
putty
terminal:
https://paste.opensuse.org/pastes/2a5c7f2fbdc4

This is exactly how I saw it on my computer, when was using
pseudo-graphics with wrong or missing locale.

The locale is there, so is it wrong?

I have no idea. It looks correct. But output looks like ncurses uses incorrect locale.

It doesn't matter if I use luit or not, they are displayed the
same. Also the ~A characters for messages with CHRS: CP437 in
the german areas are still there.

Remind me. Do you have XLAT conversion table from cp437 to cp850?

And this is still the same:

https://paste.opensuse.org/pastes/f3961b7ea085

This looks like locale is not used. I saw such pictures when
Strange.

Do you use latest GoldEd build?

Almost. I'm using this one:

commit 4b6c754756d0fa96c0c3210d6ed0b63d49ec8e6a
Author: Vitaliy Aksyonov <18148062+vitaliy-aksyonov@users.noreply.github.com>
Date: Wed Mar 6 13:38:52 2024 -0700

call setlocale() before initscr() (#86)

See section Initialization in man 3 ncurses.
https://www.man7.org/linux/man-pages/man3/ncurses.3x.html
Locale shall be initialized before ncurses initialization.

The latest one doesn't seem so change anything regarding screen
output.

Looks like incorrect locale to me again.

Do you still run it with LANG=en_US.cp850?

Yes.

In some message I saw en_EN.CP850.

The localdef command, created the directory with lowercase 'cp850' although I specified it with uppercase 'CP850'. It also shows it with lowercase 'cp' when locale -a is executed. So I switched to specifying
it as lowercase in my golded start script. But case probably doesn't matter.

No, I mean that i saw you using en_*EN*.cp850, not en_*US*.cp850. That is important.

Could you also run:
LANG=en_US.cp850 locale

Here are some tries:

wilnux5:/home/fido/log # locale
LANG=POSIX
LC_CTYPE=en_US.UTF-8

^^^^^
This is not correct

LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"

^^^^^^^
These guys too.

LC_ALL=

[...skipped...]

Make sure that all LC_-s are en_US.cp850. Especially LC_CTYPE.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-25 11:09:34, you wrote to me:

This is exactly how I saw it on my computer, when was using
pseudo-graphics with wrong or missing locale.

The locale is there, so is it wrong?

I have no idea. It looks correct. But output looks like ncurses uses incorrect
locale.

It seems so.

It doesn't matter if I use luit or not, they are displayed the
same. Also the ~A characters for messages with CHRS: CP437 in
the german areas are still there.

Remind me. Do you have XLAT conversion table from cp437 to cp850?

Yes.

In some message I saw en_EN.CP850.

The localdef command, created the directory with lowercase 'cp850'
although I specified it with uppercase 'CP850'. It also shows it
with
lowercase 'cp' when locale -a is executed. So I switched to specifying
it as lowercase in my golded start script. But case probably doesn't
matter.

No, I mean that i saw you using en_*EN*.cp850, not en_*US*.cp850. That is important.

O, sorry, I didn't notice the "_EN"...

That was what I first tried, but that made no sense, since I don't have any en_EN* locales at all (there are the en_GB* locales though).

wilnux5:/home/fido/log # locale
LANG=POSIX
LC_CTYPE=en_US.UTF-8

^^^^^
This is not correct

Ok.

Make sure that all LC_-s are en_US.cp850. Especially LC_CTYPE.

Ok that did the trick! Just setting LC_CTYPE=en_US.cp850 is enough. So now I have the following line for starting golded:

sudo -u fido LC_CTYPE=en_US.cp850 luit -encoding 'CP850' /usr/local/bin/golded -f

The linedrawing characters, and the german characters (with @CHRS: CP437) are ok too!

So this:

# sudo -u fido LC_CTYPE=en_US.cp850 locale
LANG=POSIX
LC_CTYPE=en_US.cp850
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

Seems enough to get the right output from golded!?


Bye, Wilfred.

--- FMail-lnx64 2.3.0.1-B20240319
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

25 Mar 24 21:18, you wrote to me:

Make sure that all LC_-s are en_US.cp850. Especially LC_CTYPE.

Ok that did the trick! Just setting LC_CTYPE=en_US.cp850 is enough. So now I have the following line for starting golded:

Great! Glad to help.

sudo -u fido LC_CTYPE=en_US.cp850 luit -encoding 'CP850' /usr/local/bin/golded -f

The linedrawing characters, and the german characters (with @CHRS:
CP437) are ok too!

So this:

# sudo -u fido LC_CTYPE=en_US.cp850 locale
LANG=POSIX
LC_CTYPE=en_US.cp850
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

Seems enough to get the right output from golded!?

Other LCs are pretty much not used in GoldEd, shall be fine for your case.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-25 19:03:38, you wrote to me:

Ok that did the trick! Just setting LC_CTYPE=en_US.cp850 is enough.
So now I have the following line for starting golded:

Great! Glad to help.

Yes, thank you very much!

# sudo -u fido LC_CTYPE=en_US.cp850 locale
LANG=POSIX
LC_CTYPE=en_US.cp850
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

Seems enough to get the right output from golded!?

Other LCs are pretty much not used in GoldEd, shall be fine for your case.

Pretty much? Or not at all? ;-)


On to the next issues... ;-)

I notice in my putty terminal that alignment of the columns in the area listing is off when there are unread messages in the areas:

https://paste.opensuse.org/pastes/1b0a6ac1a41e

This might not be new, and already happening before I started changing/fixing my code page issues. I'll check later if this also happens in my local konsole terminal.

Also the header with the number of messages on the message reader screen is sometimes missing a space between the first number and 'of', when I go directly to a message from the area messages listing.

https://paste.opensuse.org/pastes/80901e9650d8

Could this be related to the above issue?


Bye, Wilfred.

--- FMail-lnx64 2.3.0.1-B20240319
* Origin: FMail development HQ (2:280/464)

Hi Vitaliy,

On 2024-03-26 11:20:42, I wrote to you:

On to the next issues... ;-)

I notice in my putty terminal that alignment of the columns in the area listing is off when there are unread messages in the areas:

https://paste.opensuse.org/pastes/1b0a6ac1a41e

This might not be new, and already happening before I started changing/fixing
my code page issues. I'll check later if this also happens in my local konsole
terminal.

In my local konsole it's fine. There is a '>' character after the 0 on lines for areas where there is new mail.

Also the header with the number of messages on the message reader
screen is sometimes missing a space between the first number and
'of', when I go directly to a message from the area messages listing.

https://paste.opensuse.org/pastes/80901e9650d8

Could this be related to the above issue?

I don't see this happening in my local konsole terminal either.


This probably means I have to play with my putty settings (again)... :-/ ;-)


Bye, Wilfred.

--- FMail-lnx64 2.3.0.1-B20240319
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

26 Mar 24 18:02, you wrote to me:

On to the next issues... ;-)

I notice in my putty terminal that alignment of the columns in
the area listing is off when there are unread messages in the
areas:

https://paste.opensuse.org/pastes/1b0a6ac1a41e

This might not be new, and already happening before I started
changing/fixing my code page issues. I'll check later if this
also happens in my local konsole terminal.

In my local konsole it's fine. There is a '>' character after the 0 on lines for areas where there is new mail.

Interesting. I assume you use luit with same configuration/locale in local console and putty?

Also the header with the number of messages on the message
reader screen is sometimes missing a space between the first
number and 'of', when I go directly to a message from the area
messages listing.

https://paste.opensuse.org/pastes/80901e9650d8

Could this be related to the above issue?

I don't see this happening in my local konsole terminal either.

This probably means I have to play with my putty settings (again)...
:-/ ;-)

You full of surprises! :) Just kidding.

It may be lost in so many layers you use... I'd suggest to check if putty and local console use same/different TERM.

echo $TERM

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-26 16:17:06, you wrote to me:

In my local konsole it's fine. There is a '>' character after the 0
on lines for areas where there is new mail.

Interesting. I assume you use luit with same configuration/locale in local console and putty?

Yes.

Also the header with the number of messages on the message
reader screen is sometimes missing a space between the first
number and 'of', when I go directly to a message from the area
messages listing.

https://paste.opensuse.org/pastes/80901e9650d8

Could this be related to the above issue?

I don't see this happening in my local konsole terminal either.

This probably means I have to play with my putty settings (again)...
:-/ ;-)

You full of surprises! :) Just kidding.

Well it helped! I switched to the $TERM=putty-256color settings we discussed before, and now all is well!

It may be lost in so many layers you use... I'd suggest to check if
putty and local console use same/different TERM.

echo $TERM

Is is 'xterm' in my local konsole, and was 'linux' but now 'putty-256color' in my putty terminal.


Bye, Wilfred.

--- FMail-lnx64 2.3.0.1-B20240319
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

27 Mar 24 09:58, you wrote to me:

Also the header with the number of messages on the message
reader screen is sometimes missing a space between the first
number and 'of', when I go directly to a message from the area
messages listing.

https://paste.opensuse.org/pastes/80901e9650d8

Could this be related to the above issue?

I don't see this happening in my local konsole terminal either.

This probably means I have to play with my putty settings
(again)...
:-/ ;-)

You full of surprises! :) Just kidding.

Well it helped! I switched to the $TERM=putty-256color settings we discussed before, and now all is well!

Great. Looks like some question need to be saved in FAQ. I only never saw English version of it.

It may be lost in so many layers you use... I'd suggest to check
if putty and local console use same/different TERM.

echo $TERM

Is is 'xterm' in my local konsole, and was 'linux' but now 'putty-256color' in my putty terminal.

256color make console very different in other programs too.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-27 06:57:44, you wrote to me:

Well it helped! I switched to the $TERM=putty-256color settings we
discussed before, and now all is well!

Great. Looks like some question need to be saved in FAQ. I only never saw English version of it.

I don't think it exists.

echo $TERM

Is is 'xterm' in my local konsole, and was 'linux' but now
'putty-256color' in my putty terminal.

256color make console very different in other programs too.

I haven't tested the regular TERM=putty, but compared to TERM=linux, Midnight Commander looks the same. But my own ncurses program (configuration program for FMail tosser), looks ok now too!

https://paste.opensuse.org/pastes/c77fdc8fd4af

In the background how it looks with 'putty-256color' and how it is supposed to look. In the forground how it looks if TERM=linux.

Bye, Wilfred.

--- FMail-lnx64 2.3.0.1-B20240319
* Origin: FMail development HQ (2:280/464)

Hello Wilfred.

27 Mar 24 14:28, you wrote to me:

Well it helped! I switched to the $TERM=putty-256color settings
we discussed before, and now all is well!

Great. Looks like some question need to be saved in FAQ. I only
never saw English version of it.

I don't think it exists.

I think so. :)

echo $TERM

Is is 'xterm' in my local konsole, and was 'linux' but now
'putty-256color' in my putty terminal.

256color make console very different in other programs too.

I haven't tested the regular TERM=putty, but compared to TERM=linux, Midnight Commander looks the same. But my own ncurses program (configuration program for FMail tosser), looks ok now too!

ncurses uses terminfo and `linux` not the best for Putty. `xterm` may work, but `putty` or `putty-256color` is the best for sure.

https://paste.opensuse.org/pastes/c77fdc8fd4af

In the background how it looks with 'putty-256color' and how it is supposed to look. In the forground how it looks if TERM=linux.

putty-256color just add colors. Not every program will look different. For example `ls` will.

Vitaliy

... 640K ought to be enough for anybody
--- GoldED+/LNX 1.1.5-b20240305-beta
* Origin: Aurora, Colorado (1:104/117)

Hi Vitaliy,

On 2024-03-27 08:07:34, you wrote to me:

256color make console very different in other programs too.

I haven't tested the regular TERM=putty, but compared to TERM=linux,
Midnight Commander looks the same. But my own ncurses program
(configuration program for FMail tosser), looks ok now too!

https://paste.opensuse.org/pastes/c77fdc8fd4af

In the background how it looks with 'putty-256color' and how it is
supposed to look. In the forground how it looks if TERM=linux.

ncurses uses terminfo and `linux` not the best for Putty. `xterm` may work,
but `putty` or `putty-256color` is the best for sure.

My program works correctly in 'xterm' too.


Bye, Wilfred.

--- FMail-lnx64 2.3.0.1-B20240319
* Origin: FMail development HQ (2:280/464)