Unicode and ASCII may not be encoding correctly

Question

Hi, I am using visual studio 2022 to host a .NET 8 MVC web application project. I am trying to read a text file and write it to another text file.

   This is the code I am using:

string wdata = System.IO.File.ReadAllText(file);          

 // This is the statement that determines the encode
 // result in the output text file

 System.IO.``File.WriteAllText(wfile, wdata, System.Text.Encoding.Unicode);

  The problem is that the text file I am reading from has ”hyphens”. But when I read it, and write it to the other text file, the ”hyphens” turn into:

This is the “hyphen” in the original text I am trying to read.

This is when written using the encoding ASCII

This happens when written with encoding UTF8

Thanks for any advice or suggestions

Answer 1

Hi @iqworks Information Quality Works

Try reading the file using Encoding.UTF8 explicitly to ensure that special characters are preserved：

string wdata = System.IO.File.ReadAllText(file, System.Text.Encoding.UTF8); System.IO.File.WriteAllText(wfile, wdata, System.Text.Encoding.UTF8);

I would recommend loading the file into a text editor like Notepad++ that tells you what the encoding of the file is (in the status bar). If it's not encoded in UTF-8 to start with, reading and writing as UTF-8 won't work.

---

If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Best regards,

Rena

Answer 2

I don't think that your "hyphen" in the text file belongs to ASCII.

Please open the text file using binary editor available in Visual Studio 2022.

If your "hyphen" really belongs to ASCII it will be shown as "2D":

Answer 3

I found this fix:

// Read Text test - Read the text from a txt file and add create a new txt file

  // with the original txt file data

  public void ReadAndWriteTextTest()

  {

      // Get the root path

      var webRoot = _env.WebRootPath;

      // Text with this txt file which is a copy of the MS Word document

      // what_we_do_and_how.docm

      var inputfile = System.IO.Path.Combine(webRoot, "What_we_do_and_how ENCODED.txt");

      // The txt file name to be created.

      var outputfile = System.IO.Path.Combine(webRoot, "A6CWrite.txt");

 

      // RC@202503100000 - Get the 1252 specific code page to use with Encoding.RegisterProvider

      Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);

            

      // Read the input txt file data

      string inputdata = System.IO.File.ReadAllText(inputfile, System.Text.Encoding.GetEncoding(1252));

            

     // Create and write to the ouput txt file

      System.IO.File.WriteAllText(outputfile, inputdata, System.Text.Encoding.GetEncoding(1252));

 

 }

the outputfile now shows the correct encoding :