As this is my first attempt at a contribution towards an open source project, i am not entirely sure what you mean by docs stating that this is utf8.

The change to utf8 from ascii was a hail marry from my side, to fix an issue i had when using matplotlib together with matplotlib.rc('text', usetex=True), on my Windows machine, using MikTex as the LaTeX-interpreter.

I committed this changes in the hopes that someone would review the change and verify that i did not break any of the current functionality of the find_tex_file-function.

What the above comment by @anntzer means is that it is currently not clear if the return of subprocess.Popen(cmd, stdout=subprocess.PIPE).communicate() is necessarily utf-8 encoded, or whether it may depend on the system(-settings). If one could find a documentation where this is specified to be utf-8 in every case, this would be a good fix; else one would probably need a more sophisticated one.

@ImportanceOfBeingErnest Thank you very much for the clarification.

I will try and investigate the subprocess.Popen documentation

Another fix might be to write code that detects the language of the system, and only change the decoding when necessary.

Alright, so i have looked into the documentation of subprocess.Popen.communicate(), and found no definitive answer to what encoding it uses when piping information from the kpsewhich program.

Running kpsewhich cmmi10.tfm directly in Windows PowerShell resulted in a path C:/Users/Morten M├╕ller/AppData/Local/Programs/MiKTeX 2.9/fonts/tfm/public/cm/cmmi10.tfm , where the ø in "Morten Møller" does not appear to be decoded properly. Although, decoding the piped stdout from kpsewhich using utf-8 gives the correct path, with ø included, in the dviread.py-script.

An option which does not break any existing code, but would solve this problem is probably to try/except both cases,

try:
    res = result.decode('ascii')
except UnicodeDecodeError:
    res = result.decode('utf-8')
return res

As suggested by @ImportanceOfBeingErnest, i added the try and except error catching method to only switch to utf8-decoding when ascii fails to decode the piped info from kpsewhich.

Edit : I added this piece of code into my local version of dviread.py and it ran without errors when plotting my data. I do not know why the two checks failed on this specfic commit.

I think that sys.getfilesystemencoding() will return the system encoding which is how the return value from popen will be returned?

Unfortunately, it looks like the encoding here will depend on the kpeswitch, but the hope is that it is just passing the bytes it gets back from the filesystem through so the system encoding should be safe?

The tests fail because line 1024 contains whitespaces. Just remove them and you'll be fine.

The try... except doesn't make sense. utf-8 is a superset of ascii (or rather, it is ascii compatible), so anything that can be ascii-decoded will yield the same result when utf-8-decoded.

@tacaswell getfilesystemencoding is unlikely to be "correct" in all cases because that switched from (a default of) "mbcs" in py<3.6 to "utf-8" in py>=3.6 (PEP529), but kpathsea obviously didn't change the encoding it uses at the same time.

Note that this function already utf8-encodes the arguments to kpathsea, so it seems consistent to utf8-decode the responses. - edit: misread encode for decode.

I haven't fully unravelled the sources of kpathsea but https://tug.org/svn/texlive/trunk/Build/source/texk/kpathsea/pathsearch.c?view=markup#l38 + https://tug.org/svn/texlive/trunk/Build/source/texk/kpathsea/knj.c?revision=41586&view=markup#l407

38	#ifdef WIN32
39	#undef fputs
40	#undef puts
41	#define fputs win32_fputs
42	#define puts  win32_puts
43	#endif

407	static int __win32_fputs(const char *str, HANDLE hStdout)
408	{
409	    DWORD ret;
410	    wchar_t *wstr;
411	
412	    wstr = get_wstring_from_utf8(str, wstr=NULL);
413	
414	    if (WriteConsoleW(hStdout, wstr, wcslen(wstr), &ret, NULL) == 0) {
415	        free(wstr);
416	        return EOF;
417	    }
418	
419	    free(wstr);
420	    return ret;
421	}

plus the OP's observation that he gets paths encoded as utf-8 on Windows suggest to me that the correct encoding is utf-8 on Windows and the filesystemencoding on Unices.

@anntzer I have tested and app that does WriteConsoleW which is called with Python and find out that subprocess.Popen (subprocess.check_output too) gives empty output. So it should be printing with fputs in kpathsea_win32_fputs and since it has this File_system_codepage == CP_UTF8, the encoding must be utf8.

#include <windows.h>

int main()
{
    WriteConsoleW(GetStdHandle(STD_OUTPUT_HANDLE), L"Стдаут!\n", 8, NULL, NULL);
    WriteConsoleW(GetStdHandle(STD_ERROR_HANDLE), L"Стдерр!\n", 8, NULL, NULL);
}

C:\Temp\unicodetest>unicodetest.exe
Стдаут!
Стдерр!

import subprocess

cmd = 'unicodetest.exe'
result = subprocess.check_output(
            [cmd], stderr=subprocess.STDOUT)

print(type(result))
print('"%s"' % result)
print('"%s"' % result.decode('utf-8'))

pipe = subprocess.Popen([cmd], stdout=subprocess.PIPE)
result = pipe.communicate()[0].rstrip()

print(type(result))
print('"%s"' % result)
print('"%s"' % result.decode('utf-8'))

<class 'bytes'>
"b''"
""
Стдерр!
<class 'bytes'>
"b''"
""

My kpsewhich is from MiKTeX (kpsewhich.c)

>kpsewhich --version
MiKTeX 2.9.6600

From the prerequest list it looks like it does not depend on kpathsea, because they emulate it.

Investigate the miktex sources

kpsewhich prints content with puts

Call tree: miktex_kpathsea_path_expand -> Session::Expand -> SessionImpl::Expand -> SessionImpl::ExpandPathPatterns -> SessionImpl::ExpandPathPattern -> SessionImpl::DirectoryWalk

in directory traversing it takes entry.name:

  unique_ptr<DirectoryLister> dirLister = DirectoryLister::Open(directory, nullptr, (int)DirectoryLister::Options::DirectoriesOnly);
  DirectoryEntry entry;
  vector<PathName> subdirs;
  while (dirLister->GetNext(entry))
  {
    MIKTEX_ASSERT(entry.isDirectory);
    PathName subdir(directory);
    subdir /= entry.name;
    subdirs.push_back(subdir);
  }
  dirLister->Close();

winDirectoryLister::GetNext(DirectoryEntry & direntry)

  DirectoryEntry2 direntry2;
  if (!GetNext(direntry2))

winDirectoryLister::GetNext(DirectoryEntry2 & direntry2)

    direntry2.name = WU_(ffdat.cFileName);

WU_ macro def

#  define WU_(x) MiKTeX::Util::CharBuffer<char>(x).GetData()

CharBuffer ctor calls CharBuffer::Set

      StringUtil::CopyString(buffer, GetCapacity(), lpsz);

StringUtil::CopyString

size_t StringUtil::CopyString(char* dest, size_t destSize, const wchar_t* source)
{
  return CopyString(dest, destSize, WideCharToUTF8(source).c_str());
}

Taaadam, the output is in utf8 encoding.

Thanks for the careful investigation. Can you review #12253?

I will shortly explain the function of command_line_encoding on windows.
The function is switched by settiing of value kpse->File_system_codepage.
The value is initialized at kpathsea_set_progname() (progname.c L498) and it may set to UTF-8 in kpathsea_get_command_line_args() (knj.c L246)
If kpse->File_system_codepage == CP_UTF8, win32_fputs and win32_puts assume "input strings are encoded in UTF-8" and output Unicode characters on console by using WriteConsoleW().

Typical usage is the following:

1. get configuration of command_line_encoding from texmf.cnf (dvipdfmx.c L1030)
2. If command_line_encoding is UTF-8, get command line argument as UTF-8 and set the value kpse->File_system_codepage (dvipdfmx.c L1031)
3. run command line I/O functions via wrapper functions like win32_fputs and win32_puts

Thank you very much for your help.
If I understand correctly, the encoding actually depends on the user configuration, although I guess in practice it'll usually be utf-8?
Given that we probably don't want to parse texmf.cnf ourselves to figure out what the encoding is going to be, is there a way to force kpsewhich to output in a specific encoding (we're not picky, as long as we can know which one it is)?

In Windows,
conventional C code (ex. int main(argc, argv){}, fopen(), puts()) assumes the character encoding is a legacy encoding, CP1252 in Europe & north America, CP932 in Japan.
The function command_line_encoding provides wrapper functions to treat command line arguments, file I/O and console I/O.
If a developer wishes to fix character encoding UTF-8, the following usage will work.

• set kpse->File_system_codepage = CP_UTF8 and kpse->Is_cp932_system = 0 (ref.
  knj.c L246, knj.c L247)
• replace from POSIX functions to wrapper functions (ex. fputs -> win32_fputs)

@anntzer IIUC we can set env var command_line_encoding to utf-8 to be sure that it will use utf-8 for input and output (env var has a priority over configs) https://tug.org/svn/texlive/trunk/Build/source/texk/kpathsea/variable.c?revision=41557&view=markup#l52

@Kojoley @anntzer Does #12351 cover this problem?

I think so (to the best of my understanding...).
Thanks @MortenSHUTE for the patch and @t-tk for your explanations!

Yes. I just wanted to run tex tests on windows to be sure thing are fine, but for some reason after switching an appveyor build to TEST_ALL=yes it hanged https://ci.appveyor.com/project/Kojoley/matplotlib/builds/19205834 (and could not run locally because of my problems with needed stuff installation).