优化或删除Elixir中reduce_while调用的用法(Optimize or remove the usage of a reduce_while call in Elixir)

 我正在关注编译器上最近发布的Destroy All Software截屏视频。 所有示例代码都是用Ruby编写的，但我试图用Elixir实现相同的输出。 
 
 用简单的英语，我的方法是这样的： 
 
 
 
 将一串代码传递给递归函数（ call_tokenizer ） 
 
 
 循环遍历令牌类型，并使用当前令牌类型运行正则表达式 
 
 
 如果找到匹配项，则将结果附加到累加器，从字符串的开头删除结果，并将剩余的字符串和累加器传递给call_tokenizer 。 另外， :halt `reduce_while循环，并返回累积的标记。 
 
 
 如果没有找到结果，继续循环 
 
 
 我在Elixir中编写了一些函数，它们的工作方式是我从输入中获得预期的输出。 但是，我想我的代码至少有一个问题。 
 
 我正在使用Enum.reduce_while ，但我不认为我正在使用它，因为它打算使用它。 我的假设是有一些更好的方法可以将reduce_while为递归函数。 
 
 如果我需要在一个问题中总结一下这将是如何实现相同的结果，而不依赖于reduce_while给我的halting / continue属性？ 我应该注意这个代码示例是否还有其他问题？ 
 
 这是我的代码和预期输出： 
 
// Code
defmodule Compiler.Token do
  defstruct [:type, :value]
end

defmodule Compiler.Tokenizer do
  @token_types [
    {:def, "\\bdef\\b"},
    {:end, "\\bend\\b"},
    {:identifier, "\\b[a-zA-Z]+\\b"},
    {:integer, "\\b[0-9]+\\b"},
    {:oparen, "\\("},
    {:cparen, "\\)"}
  ]

  def tokenize() do
    code = "def f()
      1
    end"

    IO.inspect call_tokenize(code, [])
  end

  def call_tokenize("", accumulator) do
    accumulator
  end

  def call_tokenize(code, accumulator) do
    Enum.reduce_while(@token_types, "", fn {type, re}, acc ->
      result = Regex.run(~r/\A#{re}/, code)

      if result do
        value = hd(result)
        base = byte_size(value)
        token = %Compiler.Token{type: type, value: value}
        tokens = binary_part(code, base, byte_size(code) - base)
          |> String.trim()
          |> call_tokenize(accumulator ++ [token])
        {:halt, tokens}
      else
        {:cont, acc}
      end
    end)
  end
end


// Expected output
[%Compiler.Token{type: :def, value: "def"},
 %Compiler.Token{type: :identifier, value: "f"},
 %Compiler.Token{type: :oparen, value: "("},
 %Compiler.Token{type: :cparen, value: ")"},
 %Compiler.Token{type: :integer, value: "1"},
 %Compiler.Token{type: :end, value: "end"}]    

I'm following along with a recent Destroy All Software screencast on Compilers. All of the example code is written in Ruby, but I'm trying to achieve the same outputs with Elixir.
 
In plain-english, my approach goes something like this:
 
 
 
pass a string of code to a recursive function (call_tokenizer)
 
 
loop over token-types, and run a regex with the current token-type 
 
 
if a match is found, append that result to an accumulator remove the result from the beginning of the string, and pass the remaining string and the accumulator to call_tokenizer. Additionally, :halt the `reduce_while loop, and return the accumulated tokens.
 
 
if no result was found, continue on with the loop
 
 
I've written a couple functions in Elixir that work in the sense that I get the expected outputs from my input. However, I think I already have at least one issue with my code.
 
I'm using Enum.reduce_while, but I don't think I'm using it as it's intended to be used. My assumption is that there's some better way to re-write the reduce_while as a recursive function. 
 
If I needed to summarize that in a question it would be how do I achieve the same result, without relying on the halting/continue properties that reduce_while give me? Are there any other issues with this code sample that I should be aware of?
 
Here is my code and expected output:
 
// Code
defmodule Compiler.Token do
  defstruct [:type, :value]
end

defmodule Compiler.Tokenizer do
  @token_types [
    {:def, "\\bdef\\b"},
    {:end, "\\bend\\b"},
    {:identifier, "\\b[a-zA-Z]+\\b"},
    {:integer, "\\b[0-9]+\\b"},
    {:oparen, "\\("},
    {:cparen, "\\)"}
  ]

  def tokenize() do
    code = "def f()
      1
    end"

    IO.inspect call_tokenize(code, [])
  end

  def call_tokenize("", accumulator) do
    accumulator
  end

  def call_tokenize(code, accumulator) do
    Enum.reduce_while(@token_types, "", fn {type, re}, acc ->
      result = Regex.run(~r/\A#{re}/, code)

      if result do
        value = hd(result)
        base = byte_size(value)
        token = %Compiler.Token{type: type, value: value}
        tokens = binary_part(code, base, byte_size(code) - base)
          |> String.trim()
          |> call_tokenize(accumulator ++ [token])
        {:halt, tokens}
      else
        {:cont, acc}
      end
    end)
  end
end


// Expected output
[%Compiler.Token{type: :def, value: "def"},
 %Compiler.Token{type: :identifier, value: "f"},
 %Compiler.Token{type: :oparen, value: "("},
 %Compiler.Token{type: :cparen, value: ")"},
 %Compiler.Token{type: :integer, value: "1"},
 %Compiler.Token{type: :end, value: "end"}]    

原文：https://stackoverflow.com/questions/45667082