Terminal not displaying UTF-16 characters properly

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Terminal not displaying UTF-16 characters properly

Carl Hoefs
How can I set my Terminal window so it will translate UTF-16 characters properly?
TERM currently set to 'xterm-256color', but I get the following kind of output:

$ svn --version
svn, Version 1.9.7 (r1800392)

Copyright (C) 2017 The Apache Software Foundation.
This software consists of contributions made by many people;
see the NOTICE file for more information.
Subversion is open source software, see http://subversion.apache.org/

{U+4EE5}{U+4E0B}{U+306E}{U+30EA}{U+30DB}{U+30B7}{U+30C8}{U+30EA}{U+30A2}{U+30AF}{U+30BB}{U+30B9} (RA) {U+30E2}{U+30B7}{U+30E5}{U+30FC}{U+30EB}{U+304B}{U+5229}{U+7528}{U+3066}{U+304D}{U+307E}{U+3059}:

* ra_svn : svn {U+30CD}{U+30C3}{U+30C8}{U+30EF}{U+30FC}{U+30AF}{U+30D5}{U+30ED}{U+30C8}{U+30B3}{U+30EB}{U+3092}{U+4F7F}{U+3063}{U+3066}{U+30EA}{U+30DB}{U+30B7}{U+30C8}{U+30EA}{U+306B}{U+30A2}{U+30AF}{U+30BB}{U+30B9}{U+3059}{U+308B}{U+30E2}{U+30B7}{U+30E5}{U+30FC}{U+30EB}{U+3002}
  - Cyrus SASL {U+8A8D}{U+8A3C}{U+3092}{U+4F75}{U+7528}
  - 'svn' {U+30B9}{U+30AD}{U+30FC}{U+30E0}{U+3092}{U+64CD}{U+4F5C}{U+3057}{U+307E}{U+3059}
* ra_local : {U+30ED}{U+30FC}{U+30AB}{U+30EB}{U+30C6}{U+30A3}{U+30B9}{U+30AF}{U+4E0A}{U+306E}{U+30EA}{U+30DB}{U+30B7}{U+30C8}{U+30EA}{U+306B}{U+30A2}{U+30AF}{U+30BB}{U+30B9}{U+3059}{U+308B}{U+30E2}{U+30B7}{U+30E5}{U+30FC}{U+30EB}{U+3002}
  - 'file' {U+30B9}{U+30AD}{U+30FC}{U+30E0}{U+3092}{U+64CD}{U+4F5C}{U+3057}{U+307E}{U+3059}
* ra_serf : Module for accessing a repository via WebDAV protocol using serf.
  - using serf 1.3.8 (compiled with 1.3.8)
  - 'http' {U+30B9}{U+30AD}{U+30FC}{U+30E0}{U+3092}{U+64CD}{U+4F5C}{U+3057}{U+307E}{U+3059}
  - 'https' {U+30B9}{U+30AD}{U+30FC}{U+30E0}{U+3092}{U+64CD}{U+4F5C}{U+3057}{U+307E}{U+3059}

_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk
Reply | Threaded
Open this post in threaded view
|

Re: Terminal not displaying UTF-16 characters properly

Macs R We
Aaargh.  You'd do better asking this on a *nix list.

> On Jan 17, 2018, at 9:04 PM, Carl Hoefs <[hidden email]> wrote:
>
> How can I set my Terminal window so it will translate UTF-16 characters properly?
> TERM currently set to 'xterm-256color', but I get the following kind of output:
>
> $ svn --version
> svn, Version 1.9.7 (r1800392)
>
> Copyright (C) 2017 The Apache Software Foundation.
> This software consists of contributions made by many people;
> see the NOTICE file for more information.
> Subversion is open source software, see http://subversion.apache.org/
>
> {U+4EE5}{U+4E0B}{U+306E}{U+30EA}{U+30DB}{U+30B7}{U+30C8}{U+30EA}{U+30A2}{U+30AF}{U+30BB}{U+30B9} (RA) {U+30E2}{U+30B7}{U+30E5}{U+30FC}{U+30EB}{U+304B}{U+5229}{U+7528}{U+3066}{U+304D}{U+307E}{U+3059}:
>
> * ra_svn : svn {U+30CD}{U+30C3}{U+30C8}{U+30EF}{U+30FC}{U+30AF}{U+30D5}{U+30ED}{U+30C8}{U+30B3}{U+30EB}{U+3092}{U+4F7F}{U+3063}{U+3066}{U+30EA}{U+30DB}{U+30B7}{U+30C8}{U+30EA}{U+306B}{U+30A2}{U+30AF}{U+30BB}{U+30B9}{U+3059}{U+308B}{U+30E2}{U+30B7}{U+30E5}{U+30FC}{U+30EB}{U+3002}
>  - Cyrus SASL {U+8A8D}{U+8A3C}{U+3092}{U+4F75}{U+7528}
>  - 'svn' {U+30B9}{U+30AD}{U+30FC}{U+30E0}{U+3092}{U+64CD}{U+4F5C}{U+3057}{U+307E}{U+3059}
> * ra_local : {U+30ED}{U+30FC}{U+30AB}{U+30EB}{U+30C6}{U+30A3}{U+30B9}{U+30AF}{U+4E0A}{U+306E}{U+30EA}{U+30DB}{U+30B7}{U+30C8}{U+30EA}{U+306B}{U+30A2}{U+30AF}{U+30BB}{U+30B9}{U+3059}{U+308B}{U+30E2}{U+30B7}{U+30E5}{U+30FC}{U+30EB}{U+3002}
>  - 'file' {U+30B9}{U+30AD}{U+30FC}{U+30E0}{U+3092}{U+64CD}{U+4F5C}{U+3057}{U+307E}{U+3059}
> * ra_serf : Module for accessing a repository via WebDAV protocol using serf.
>  - using serf 1.3.8 (compiled with 1.3.8)
>  - 'http' {U+30B9}{U+30AD}{U+30FC}{U+30E0}{U+3092}{U+64CD}{U+4F5C}{U+3057}{U+307E}{U+3059}
>  - 'https' {U+30B9}{U+30AD}{U+30FC}{U+30E0}{U+3092}{U+64CD}{U+4F5C}{U+3057}{U+307E}{U+3059}
>
> _______________________________________________
> MacOSX-talk mailing list
> [hidden email]
> http://www.omnigroup.com/mailman/listinfo/macosx-talk

_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk
Reply | Threaded
Open this post in threaded view
|

Re: Terminal not displaying UTF-16 characters properly

@lbutlr
In reply to this post by Carl Hoefs

On 17 Jan 2018, at 21:04, Carl Hoefs <[hidden email]> wrote:
>
> How can I set my Terminal window so it will translate UTF-16 characters properly?
> TERM currently set to 'xterm-256color', but I get the following kind of output:

UTF-8 is macOS native encoding. I don't think UTF-16 works well at all, and is not a default available encoding for the Terminal application. However, if you go to "Preferences -> Encodings" you can enable UTF-16. I wouldn't expect this to work well, but I've never tried it.

If you've already done this... well, I can't say I am shocked.

If I set my terminal to UTF-16 everything becomes Chinese characters.

䱡獴潧楮㨠坥搠䩡渠ㄷ′㌺㔳㨲㘠潮⁴瑹猰〳ഊ᭝㜻晩汥㨯⽓湯眮汯捡氯啳敲猯汢畴汲ܛ嬿㄰㌴栛嬿㄰㌴栛嬿㄰㌴栛嬰㬳㍭卮潷㨛嬰㬳㉭縯᭛ねഊ᭝㜻晩汥㨯⽓湯眮汯捡氯啳敲猯汢畴汲᭛〻㌳浓湯眺᭛〻㌲浾⼛嬰洠ഊ᭝㜻晩汥㨯⽓湯眮汯捡氯啳敲猯汢畴汲᭛〻㌳浓湯眺᭛〻㌲浾⼛嬰洠ഊ᭝㜻晩汥㨯⽓湯眮汯捡氯啳敲猯汢畴汲᭛〻㌳浓湯眺᭛〻㌲浾⼛嬰洠ഊ᭝㜻晩汥㨯⽓湯眮汯捡氯啳敲猯汢畴汲᭛〻㌳浓湯眺᭛〻㌲浾⼛嬰洠


--
"But you read a lot of books, I'm thinking. Hard to have faith, ain't
it, when you've read too many books?"

_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk
Reply | Threaded
Open this post in threaded view
|

Re: Terminal not displaying UTF-16 characters properly

Jean-Christophe Helary


> On Jan 18, 2018, at 16:53, @lbutlr <[hidden email]> wrote:
>
> If I set my terminal to UTF-16 everything becomes Chinese characters.

Not all Chinese :)
Some Malayan, some Japanese, some Balinese...

> 䱡獴潧楮㨠坥搠䩡渠ㄷ′㌺㔳㨲㘠潮⁴瑹猰〳ഊ᭝㜻晩汥㨯⽓湯眮汯捡氯啳敲猯汢畴汲ܛ嬿㄰㌴栛嬿㄰㌴栛嬿㄰㌴栛嬰㬳㍭卮潷㨛嬰㬳㉭縯᭛ねഊ᭝㜻晩汥㨯⽓湯眮汯捡氯啳敲猯汢畴汲᭛〻㌳浓湯眺᭛〻㌲浾⼛嬰洠ഊ᭝㜻晩汥㨯⽓湯眮汯捡氯啳敲猯汢畴汲᭛〻㌳浓湯眺᭛〻㌲浾⼛嬰洠ഊ᭝㜻晩汥㨯⽓湯眮汯捡氯啳敲猯汢畴汲᭛〻㌳浓湯眺᭛〻㌲浾⼛嬰洠ഊ᭝㜻晩汥㨯⽓湯眮汯捡氯啳敲猯汢畴汲᭛〻㌳浓湯眺᭛〻㌲浾⼛嬰洠

Jean-Christophe Helary
-----------------------------------------------
@brandelune http://mac4translators.blogspot.com


_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk
Reply | Threaded
Open this post in threaded view
|

Re: Terminal not displaying UTF-16 characters properly

Carl Hoefs
In reply to this post by @lbutlr

> On Jan 18, 2018, at 12:53 AM, @lbutlr <[hidden email]> wrote:
>
>
> On 17 Jan 2018, at 21:04, Carl Hoefs <[hidden email]> wrote:
>>
>> How can I set my Terminal window so it will translate UTF-16 characters properly?
>> TERM currently set to 'xterm-256color', but I get the following kind of output:
>
> UTF-8 is macOS native encoding. I don't think UTF-16 works well at all, and is not a default available encoding for the Terminal application. However, if you go to "Preferences -> Encodings" you can enable UTF-16. I wouldn't expect this to work well, but I've never tried it.
>
> If you've already done this... well, I can't say I am shocked.
>
> If I set my terminal to UTF-16 everything becomes Chinese characters.
>
> 䱡獴潧楮㨠坥搠䩡渠ㄷ′㌺㔳㨲㘠潮⁴瑹猰〳ഊ᭝㜻晩汥㨯⽓湯眮汯捡氯啳敲猯汢畴汲ܛ嬿㄰㌴栛嬿㄰㌴栛嬿㄰㌴栛嬰㬳㍭卮潷㨛嬰㬳㉭縯᭛ねഊ᭝㜻晩汥㨯⽓湯眮汯捡氯啳敲猯汢畴汲᭛〻㌳浓湯眺᭛〻㌲浾⼛嬰洠ഊ᭝㜻晩汥㨯⽓湯眮汯捡氯啳敲猯汢畴汲᭛〻㌳浓湯眺᭛〻㌲浾⼛嬰洠ഊ᭝㜻晩汥㨯⽓湯眮汯捡氯啳敲猯汢畴汲᭛〻㌳浓湯眺᭛〻㌲浾⼛嬰洠ഊ᭝㜻晩汥㨯⽓湯眮汯捡氯啳敲猯汢畴汲᭛〻㌳浓湯眺᭛〻㌲浾⼛嬰洠

Yeah, that’s gibberish. The odd thing is that this seems to be only happening with Subversion. So maybe it isn’t a Terminal thing, per se.

$ svn info
svn: E155007: '/Users/carl' {U+306F}{U+4F5C}{U+696D}{U+30B3}{U+30D2}{U+30FC}{U+3066}{U+306F}{U+3042}{U+308A}{U+307E}{U+305B}{U+3093}

The above is supposed to say:

svn: E155007: '/Users/carl’ is not a working copy

…but I don’t know what flavor of UTF to use. There aren’t even enough UTF-16 characters in the above line to match.
I’m confused...

-Carl



_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk
Reply | Threaded
Open this post in threaded view
|

Re: Terminal not displaying UTF-16 characters properly

Jean-Christophe Helary

> On Jan 19, 2018, at 2:15, Carl Hoefs <[hidden email]> wrote:
> Subversion. So maybe it isn’t a Terminal thing, per se.
>
> $ svn info
> svn: E155007: '/Users/carl' {U+306F}{U+4F5C}{U+696D}{U+30B3}{U+30D2}{U+30FC}{U+3066}{U+306F}{U+3042}{U+308A}{U+307E}{U+305B}{U+3093}
>
> The above is supposed to say:
>
> svn: E155007: '/Users/carl’ is not a working copy

Mystery solved.

If you check the characters output by subversion you get:

{U+306F} は
{U+4F5C} 作
{U+696D} 業
{U+30B3} コ
{U+30D2} ヒ
{U+30FC} ー
{U+3066} て
{U+306F} は
{U+3042} あ
{U+308A} り
{U+307E} ま
{U+305B} せ
{U+3093} ん

From https://codepoints.net

Which reads:

'/Users/carl' は作業コヒーてはありません

it actually should be:

'/Users/carl' は作業コピーではありません

but I suspect there are representation issues for compound glyphs and I just woke up so that part is left for later.

Basically it is the Japanese way of saying:

'/Users/carl’ is not a working copy

So svn is talking Japanese to you. It is not an encoding issue, it is a language setting issue.


Jean-Christophe Helary
-----------------------------------------------
@brandelune http://mac4translators.blogspot.com


_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk
Reply | Threaded
Open this post in threaded view
|

Re: Terminal not displaying UTF-16 characters properly

Carl Hoefs

> On Jan 18, 2018, at 4:51 PM, Jean-Christophe Helary <[hidden email]> wrote:
>
>
>> On Jan 19, 2018, at 2:15, Carl Hoefs <[hidden email]> wrote:
>> Subversion. So maybe it isn’t a Terminal thing, per se.
>>
>> $ svn info
>> svn: E155007: '/Users/carl' {U+306F}{U+4F5C}{U+696D}{U+30B3}{U+30D2}{U+30FC}{U+3066}{U+306F}{U+3042}{U+308A}{U+307E}{U+305B}{U+3093}
>>
>> The above is supposed to say:
>>
>> svn: E155007: '/Users/carl’ is not a working copy
>
> Mystery solved.
>
> If you check the characters output by subversion you get:
>
> {U+306F} は
> {U+4F5C} 作
> {U+696D} 業
> {U+30B3} コ
> {U+30D2} ヒ
> {U+30FC} ー
> {U+3066} て
> {U+306F} は
> {U+3042} あ
> {U+308A} り
> {U+307E} ま
> {U+305B} せ
> {U+3093} ん
>
> From https://codepoints.net
>
> Which reads:
>
> '/Users/carl' は作業コヒーてはありません
>
> it actually should be:
>
> '/Users/carl' は作業コピーではありません
>
> but I suspect there are representation issues for compound glyphs and I just woke up so that part is left for later.
>
> Basically it is the Japanese way of saying:
>
> '/Users/carl’ is not a working copy
>
> So svn is talking Japanese to you. It is not an encoding issue, it is a language setting issue.

Well now, that there’s the gall-darned craziest thing I ever did see! Japanese?!

I noticed that the LANG env var wasn’t set in the shell I was using. Setting it to “en_US.UTF-8” restores sanity.

Thanks for the sleuthing. I never would have guessed.

-Carl

_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk
Reply | Threaded
Open this post in threaded view
|

Re: Terminal not displaying UTF-16 characters properly

Jean-Christophe Helary


> On Jan 19, 2018, at 9:42, Carl Hoefs <[hidden email]> wrote:
>
>>
>> On Jan 18, 2018, at 4:51 PM, Jean-Christophe Helary <[hidden email]> wrote:
>>
>>
>>> On Jan 19, 2018, at 2:15, Carl Hoefs <[hidden email]> wrote:
>>> Subversion. So maybe it isn’t a Terminal thing, per se.
>>>
>>> $ svn info
>>> svn: E155007: '/Users/carl' {U+306F}{U+4F5C}{U+696D}{U+30B3}{U+30D2}{U+30FC}{U+3066}{U+306F}{U+3042}{U+308A}{U+307E}{U+305B}{U+3093}
>>>
>>> The above is supposed to say:
>>>
>>> svn: E155007: '/Users/carl’ is not a working copy

> Well now, that there’s the gall-darned craziest thing I ever did see! Japanese?!

I thought it was weird from the outset because the number and pattern of characters don't match. For ex, there should have been 4 characters for [space] but there was no pattern like that in the output.

Also, I happen to translate/localize to and from Japanese, so that was a big help. :)

Jean-Christophe Helary
-----------------------------------------------
@brandelune http://mac4translators.blogspot.com


_______________________________________________
MacOSX-talk mailing list
[hidden email]
http://www.omnigroup.com/mailman/listinfo/macosx-talk