From b3d5ee4f89e60df0da877cdef9207db166216ff6 Mon Sep 17 00:00:00 2001 From: Shuaichen Chang Date: Thu, 12 Sep 2019 14:57:14 -0400 Subject: [PATCH 01/42] add auxiliary task result --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 5ea0694..24396f3 100755 --- a/README.md +++ b/README.md @@ -49,6 +49,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | | [SQLova
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 81.6 | 87.2 | 80.7 | 86.2 | | | [IncSQL
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 49.9 | 84.0 | 49.9 | 83.7 | | +| [Auxiliary Mapping Task
(Chang 2019)](https://arxiv.org/pdf/1908.11052.pdf) | 75.6 | 81.9 | 75.0 | 81.7 | | | [MQAN (unordered)
(McCann 2018)](https://arxiv.org/abs/1806.08730) | 76.1 | 82.0 | 75.4 | 81.4 | | | [MQAN (ordered)
(McCann 2018)](https://arxiv.org/abs/1806.08730) | 73.5 | 82.0 | 73.2 | 81.4 | | | [Coarse2Fine
(Dong 2018)](https://arxiv.org/abs/1805.04793) | 72.5 | 79.0 | 71.7 | 78.5 | | From f0da9cc4f9f98233ce7d1a9477932c9412d9cf85 Mon Sep 17 00:00:00 2001 From: Shuaichen Chang Date: Thu, 12 Sep 2019 15:04:57 -0400 Subject: [PATCH 02/42] add auxiliary task result --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 24396f3..d2ea611 100755 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | | [SQLova
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 81.6 | 87.2 | 80.7 | 86.2 | | | [IncSQL
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 49.9 | 84.0 | 49.9 | 83.7 | | -| [Auxiliary Mapping Task
(Chang 2019)](https://arxiv.org/pdf/1908.11052.pdf) | 75.6 | 81.9 | 75.0 | 81.7 | | +| [Auxiliary Mapping Task
(Chang 2019)](https://arxiv.org/pdf/1908.11052.pdf) | 76.0 | 82.3 | 75.0 | 81.7 | | | [MQAN (unordered)
(McCann 2018)](https://arxiv.org/abs/1806.08730) | 76.1 | 82.0 | 75.4 | 81.4 | | | [MQAN (ordered)
(McCann 2018)](https://arxiv.org/abs/1806.08730) | 73.5 | 82.0 | 73.2 | 81.4 | | | [Coarse2Fine
(Dong 2018)](https://arxiv.org/abs/1805.04793) | 72.5 | 79.0 | 71.7 | 78.5 | | From ad04d06255abcef704849e080e572cafd4bca7e6 Mon Sep 17 00:00:00 2001 From: blwang Date: Fri, 20 Sep 2019 21:18:17 +0100 Subject: [PATCH 03/42] add an entry of latent alignment model --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 5ea0694..5625610 100755 --- a/README.md +++ b/README.md @@ -36,6 +36,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [Rule-SQL (Guo 2019)](https://arxiv.org/abs/1907.00620) | 61.1 +/- 0.2 | 61.0 +/- 0.3 | | [MAPO (Liang 2018)](https://arxiv.org/abs/1807.02322) | 72.2 +/- 0.2 | 72.1 +/- 0.3 | | [MeRL (Agarwal 2019)](https://arxiv.org/abs/1902.07198) | 74.9 +/- 0.1 | 74.8 +/- 0.2 | +| [LatentAlignment (Wang 2019)](https://arxiv.org/abs/1909.04165) | 79.4 | 79.3 | | Anonymous (2019) | 84.4 | 83.9 | ### Supervised via logical forms From 1f3c6e63825e107ca9336a2eb55c645b4816588d Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Mon, 27 Jan 2020 15:54:11 -0800 Subject: [PATCH 04/42] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 5ea0694..d3ac6e3 100755 --- a/README.md +++ b/README.md @@ -46,6 +46,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | | [Execution-Guided Decoding
(Wang 2018)](https://arxiv.org/abs/1807.03100) | 76.0 | 84.0 | 75.4 | 83.8 | Inference | +| HydraNet (Anonymous 2020) | 83.6 | 88.9 | 83.8 | 89.0 | | | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | | [SQLova
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 81.6 | 87.2 | 80.7 | 86.2 | | | [IncSQL
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 49.9 | 84.0 | 49.9 | 83.7 | | From 350f2bf50510f090db1cc7516e1ebdccf1f8fa6c Mon Sep 17 00:00:00 2001 From: Tong Guo <779222056@qq.com> Date: Fri, 7 Feb 2020 22:43:35 +0800 Subject: [PATCH 05/42] Update README.md --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 762e456..01cbd77 100755 --- a/README.md +++ b/README.md @@ -44,9 +44,11 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | +| [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | | [Execution-Guided Decoding
(Wang 2018)](https://arxiv.org/abs/1807.03100) | 76.0 | 84.0 | 75.4 | 83.8 | Inference | +| [(Guo 2019)
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^ | 84.3 | 90.3 | 83.7 | 89.2 | | | HydraNet (Anonymous 2020) | 83.6 | 88.9 | 83.8 | 89.0 | | | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | | [SQLova
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 81.6 | 87.2 | 80.7 | 86.2 | | From 9368822340075fdcf41370b9106f2277cf745cbd Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Fri, 7 Feb 2020 10:14:25 -0800 Subject: [PATCH 06/42] Update README.md --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index d3ac6e3..f550ea7 100755 --- a/README.md +++ b/README.md @@ -42,11 +42,12 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | +| HydraNet + Execution-Guided Decoding (Anonymous 2020)^ | 86.6 | 92.1 | 86.5 | 92.0 | | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | | [Execution-Guided Decoding
(Wang 2018)](https://arxiv.org/abs/1807.03100) | 76.0 | 84.0 | 75.4 | 83.8 | Inference | -| HydraNet (Anonymous 2020) | 83.6 | 88.9 | 83.8 | 89.0 | | +| HydraNet (Anonymous 2020)^ | 83.6 | 88.9 | 83.8 | 89.0 | | | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | | [SQLova
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 81.6 | 87.2 | 80.7 | 86.2 | | | [IncSQL
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 49.9 | 84.0 | 49.9 | 83.7 | | From 8cd780fae006527e5cf28b37e8d6769b795e7b24 Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Fri, 7 Feb 2020 11:10:33 -0800 Subject: [PATCH 07/42] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f550ea7..67def92 100755 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | -| HydraNet + Execution-Guided Decoding (Anonymous 2020)^ | 86.6 | 92.1 | 86.5 | 92.0 | | +| HydraNet + Execution-Guided Decoding (Anonymous 2020)^ | 86.6 | 92.1 | 86.5 | 92.0 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | From 7ef17e188b01ef398b91ce8d70ae6ea1deeab9d1 Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Fri, 7 Feb 2020 17:29:03 -0800 Subject: [PATCH 08/42] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 67def92..678f1ef 100755 --- a/README.md +++ b/README.md @@ -42,12 +42,12 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | -| HydraNet + Execution-Guided Decoding (Anonymous 2020)^ | 86.6 | 92.1 | 86.5 | 92.0 | Inference | +| HydraNet + Execution-Guided Decoding (Anonymous 2020) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | | [Execution-Guided Decoding
(Wang 2018)](https://arxiv.org/abs/1807.03100) | 76.0 | 84.0 | 75.4 | 83.8 | Inference | -| HydraNet (Anonymous 2020)^ | 83.6 | 88.9 | 83.8 | 89.0 | | +| HydraNet (Anonymous 2020) | 83.6 | 89.1 | 83.8 | 89.2 | | | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | | [SQLova
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 81.6 | 87.2 | 80.7 | 86.2 | | | [IncSQL
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 49.9 | 84.0 | 49.9 | 83.7 | | From 56e340bc91db78477ede379b66920d9a6d47cdfd Mon Sep 17 00:00:00 2001 From: Victor Zhong Date: Wed, 19 Feb 2020 15:05:12 -0800 Subject: [PATCH 09/42] Update README.md --- README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 024cf71..4a547a7 100755 --- a/README.md +++ b/README.md @@ -33,11 +33,12 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev execution accuracy | Test execution accuracy | | :---: | :---: | :---: | -| [Rule-SQL (Guo 2019)](https://arxiv.org/abs/1907.00620) | 61.1 +/- 0.2 | 61.0 +/- 0.3 | -| [MAPO (Liang 2018)](https://arxiv.org/abs/1807.02322) | 72.2 +/- 0.2 | 72.1 +/- 0.3 | -| [MeRL (Agarwal 2019)](https://arxiv.org/abs/1902.07198) | 74.9 +/- 0.1 | 74.8 +/- 0.2 | +| [HardEM (Min 2019)](https://arxiv.org/abs/1909.04849) | 84.4 | 83.9 | | [LatentAlignment (Wang 2019)](https://arxiv.org/abs/1909.04165) | 79.4 | 79.3 | -| Anonymous (2019) | 84.4 | 83.9 | +| [MeRL (Agarwal 2019)](https://arxiv.org/abs/1902.07198) | 74.9 +/- 0.1 | 74.8 +/- 0.2 | +| [MAPO (Liang 2018)](https://arxiv.org/abs/1807.02322) | 72.2 +/- 0.2 | 72.1 +/- 0.3 | +| [Rule-SQL (Guo 2019)](https://arxiv.org/abs/1907.00620) | 61.1 +/- 0.2 | 61.0 +/- 0.3 | + ### Supervised via logical forms From 1aa1a3bbe36ece7762cd56afdfd9994b8b2afaf9 Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Wed, 11 Mar 2020 15:31:18 -0700 Subject: [PATCH 10/42] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 024cf71..5aed203 100755 --- a/README.md +++ b/README.md @@ -43,14 +43,14 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | -| HydraNet + Execution-Guided Decoding (Anonymous 2020) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | +| [HydraNet
+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/) (Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | | [Execution-Guided Decoding
(Wang 2018)](https://arxiv.org/abs/1807.03100) | 76.0 | 84.0 | 75.4 | 83.8 | Inference | +| [HydraNet (Lyu 2020)] (https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/) (Microsoft Dynamics 365 AI) | 83.6 | 89.1 | 83.8 | 89.2 | | | [(Guo 2019)
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^ | 84.3 | 90.3 | 83.7 | 89.2 | | -| HydraNet (Anonymous 2020) | 83.6 | 89.1 | 83.8 | 89.2 | | | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | | [SQLova
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 81.6 | 87.2 | 80.7 | 86.2 | | | [IncSQL
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 49.9 | 84.0 | 49.9 | 83.7 | | From b498bcd97be20705134f870245bce0daf3eeee10 Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Wed, 11 Mar 2020 15:32:59 -0700 Subject: [PATCH 11/42] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5aed203..e42d0a1 100755 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | | [Execution-Guided Decoding
(Wang 2018)](https://arxiv.org/abs/1807.03100) | 76.0 | 84.0 | 75.4 | 83.8 | Inference | -| [HydraNet (Lyu 2020)] (https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/) (Microsoft Dynamics 365 AI) | 83.6 | 89.1 | 83.8 | 89.2 | | +| [HydraNet
(Lyu 2020)] (https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/) (Microsoft Dynamics 365 AI) | 83.6 | 89.1 | 83.8 | 89.2 | | | [(Guo 2019)
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^ | 84.3 | 90.3 | 83.7 | 89.2 | | | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | | [SQLova
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 81.6 | 87.2 | 80.7 | 86.2 | | From c6fb1521ffd74ef4c4ad48b3e3313c5769bf6446 Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Wed, 11 Mar 2020 15:35:18 -0700 Subject: [PATCH 12/42] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e42d0a1..5aed203 100755 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | | [Execution-Guided Decoding
(Wang 2018)](https://arxiv.org/abs/1807.03100) | 76.0 | 84.0 | 75.4 | 83.8 | Inference | -| [HydraNet
(Lyu 2020)] (https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/) (Microsoft Dynamics 365 AI) | 83.6 | 89.1 | 83.8 | 89.2 | | +| [HydraNet (Lyu 2020)] (https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/) (Microsoft Dynamics 365 AI) | 83.6 | 89.1 | 83.8 | 89.2 | | | [(Guo 2019)
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^ | 84.3 | 90.3 | 83.7 | 89.2 | | | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | | [SQLova
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 81.6 | 87.2 | 80.7 | 86.2 | | From fdd1948c6522322cc9f31371a4f6cf23fb9e04d7 Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Wed, 11 Mar 2020 15:36:27 -0700 Subject: [PATCH 13/42] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5aed203..611aac7 100755 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | | [Execution-Guided Decoding
(Wang 2018)](https://arxiv.org/abs/1807.03100) | 76.0 | 84.0 | 75.4 | 83.8 | Inference | -| [HydraNet (Lyu 2020)] (https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/) (Microsoft Dynamics 365 AI) | 83.6 | 89.1 | 83.8 | 89.2 | | +| [HydraNet (Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/) (Microsoft Dynamics 365 AI) | 83.6 | 89.1 | 83.8 | 89.2 | | | [(Guo 2019)
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^ | 84.3 | 90.3 | 83.7 | 89.2 | | | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | | [SQLova
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 81.6 | 87.2 | 80.7 | 86.2 | | From 2f03978012596e44e887570bbe4e330dea4aab8d Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Wed, 11 Mar 2020 15:37:41 -0700 Subject: [PATCH 14/42] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 611aac7..38ecf94 100755 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | -| [HydraNet
+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/) (Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | +| HydraNet + Execution-Guided Decoding [(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/) (Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | From 0a32a786d74397478abeb83646443981fdedd67c Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Wed, 11 Mar 2020 15:38:43 -0700 Subject: [PATCH 15/42] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 38ecf94..07f8dec 100755 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | -| HydraNet + Execution-Guided Decoding [(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/) (Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | +| [HydraNet + Execution-Guided Decoding (Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/) (Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | From c52e8b385f43a044d763b045d136da8fa15d1718 Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Wed, 11 Mar 2020 15:39:42 -0700 Subject: [PATCH 16/42] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 07f8dec..68fea40 100755 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | -| [HydraNet + Execution-Guided Decoding (Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/) (Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | +| [HydraNet + Execution-Guided Decoding
(Lyu 2020)
](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI)
| 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | From 3f0fa13b1f5d2072c398aa9cb5c6052ef41d1d51 Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Wed, 11 Mar 2020 15:40:18 -0700 Subject: [PATCH 17/42] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 68fea40..f5dc117 100755 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | -| [HydraNet + Execution-Guided Decoding
(Lyu 2020)
](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI)
| 86.6 | 92.4 | 86.5 | 92.2 | Inference | +| [HydraNet + Execution-Guided Decoding
(Lyu 2020)
](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | From 8c58a9946e5c220aee5f61ed4dc860c7491cd4da Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Wed, 11 Mar 2020 15:40:54 -0700 Subject: [PATCH 18/42] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f5dc117..8fca651 100755 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | -| [HydraNet + Execution-Guided Decoding
(Lyu 2020)
](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | +| [HydraNet + Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | From e7ce0bad3a833e88cf482bcf1bb4396625f78072 Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Wed, 11 Mar 2020 15:41:46 -0700 Subject: [PATCH 19/42] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8fca651..48af15a 100755 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | | [Execution-Guided Decoding
(Wang 2018)](https://arxiv.org/abs/1807.03100) | 76.0 | 84.0 | 75.4 | 83.8 | Inference | -| [HydraNet (Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/) (Microsoft Dynamics 365 AI) | 83.6 | 89.1 | 83.8 | 89.2 | | +| [HydraNet (Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 83.6 | 89.1 | 83.8 | 89.2 | | | [(Guo 2019)
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^ | 84.3 | 90.3 | 83.7 | 89.2 | | | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | | [SQLova
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 81.6 | 87.2 | 80.7 | 86.2 | | From d8e19bfe1df4b97dd2cf5f394907750ba188b484 Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Wed, 11 Mar 2020 15:42:52 -0700 Subject: [PATCH 20/42] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 48af15a..d3c41a9 100755 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | -| [HydraNet + Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | +| [HydraNet+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | From d0f6c4ee25562cd40e3617bef9689f84abc04266 Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Wed, 11 Mar 2020 15:43:35 -0700 Subject: [PATCH 21/42] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d3c41a9..ced6392 100755 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | -| [HydraNet+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | +| [HydraNet+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | From 71c0cd593c19051a4db14dad14aad59897a8a151 Mon Sep 17 00:00:00 2001 From: Souvik Kundu Date: Wed, 11 Mar 2020 15:44:17 -0700 Subject: [PATCH 22/42] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ced6392..b56ca8d 100755 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | -| [HydraNet+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | +| [HydraNet
+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | From f9c15bf3865563d8423bcb2a26b99b48d3d7717c Mon Sep 17 00:00:00 2001 From: Devanshu Desai <37163544+devanshuDesai@users.noreply.github.com> Date: Sun, 29 Mar 2020 00:36:47 -0700 Subject: [PATCH 23/42] fixed minor typo --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 87a724a..7aa9c4e 100755 --- a/README.md +++ b/README.md @@ -116,7 +116,7 @@ These files are contained in the `*.jsonl` files. A line looks like the followin The fields represent the following: -- `phase`: the phase in which the dataset was collection. We collected WikiSQL in two phases. +- `phase`: the phase in which the dataset was collected. We collected WikiSQL in two phases. - `question`: the natural language question written by the worker. - `table_id`: the ID of the table to which this question is addressed. - `sql`: the SQL query corresponding to the question. This has the following subfields: From 8ee1516a7dd19774754dc4b15b4f1e1774836197 Mon Sep 17 00:00:00 2001 From: iesql <68574828+iesql@users.noreply.github.com> Date: Tue, 21 Jul 2020 10:15:13 +0800 Subject: [PATCH 24/42] add IE-SQL (+EG) result --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 87a724a..2bf4410 100755 --- a/README.md +++ b/README.md @@ -44,6 +44,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | +| [IE-SQL
+Execution-Guided Decoding
(anonymous)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
| 87.9 | 92.6 | 87.8 | 92.5 | Inference | | [HydraNet
+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | From 6bbfc1ced4b3edc97e25c7e8c4821f48da9b56f9 Mon Sep 17 00:00:00 2001 From: iesql <68574828+iesql@users.noreply.github.com> Date: Tue, 21 Jul 2020 10:18:22 +0800 Subject: [PATCH 25/42] add IE-SQL result (without EG) --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 2bf4410..33179db 100755 --- a/README.md +++ b/README.md @@ -52,6 +52,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | | [Execution-Guided Decoding
(Wang 2018)](https://arxiv.org/abs/1807.03100) | 76.0 | 84.0 | 75.4 | 83.8 | Inference | | [HydraNet (Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 83.6 | 89.1 | 83.8 | 89.2 | | +| [IE-SQL (anonymous)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
| 84.6 | 88.7 | 84.6 | 88.8 | | | [(Guo 2019)
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^ | 84.3 | 90.3 | 83.7 | 89.2 | | | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | | [SQLova
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 81.6 | 87.2 | 80.7 | 86.2 | | From abe18fe3ee6e9035f8e2cef5ecb3ca6a44e75838 Mon Sep 17 00:00:00 2001 From: iesql <68574828+iesql@users.noreply.github.com> Date: Mon, 3 Aug 2020 19:15:47 +0800 Subject: [PATCH 26/42] Update README.md updated the author and affiliation --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 33179db..e294123 100755 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | -| [IE-SQL
+Execution-Guided Decoding
(anonymous)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
| 87.9 | 92.6 | 87.8 | 92.5 | Inference | +| [IE-SQL
+Execution-Guided Decoding
(Ma 2020)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
(Ping An Life Insurance, AI Team)| 87.9 | 92.6 | 87.8 | 92.5 | Inference | | [HydraNet
+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | From e7b909fee7a04e828be8809d5fb76bad7f91165a Mon Sep 17 00:00:00 2001 From: iesql <68574828+iesql@users.noreply.github.com> Date: Mon, 3 Aug 2020 19:24:30 +0800 Subject: [PATCH 27/42] Update README.md - updated author and affiliation for IE-SQL (without EG) - fix a previous error: lower the position of IE-SQL (without EG) by 1 rank. --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index e294123..da55023 100755 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | -| [IE-SQL
+Execution-Guided Decoding
(Ma 2020)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
(Ping An Life Insurance, AI Team)| 87.9 | 92.6 | 87.8 | 92.5 | Inference | +| [IE-SQL
+Execution-Guided Decoding
(Ma 2020)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
(Ping An Life, AI Team)| 87.9 | 92.6 | 87.8 | 92.5 | Inference | | [HydraNet
+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | @@ -52,8 +52,8 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | | [Execution-Guided Decoding
(Wang 2018)](https://arxiv.org/abs/1807.03100) | 76.0 | 84.0 | 75.4 | 83.8 | Inference | | [HydraNet (Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 83.6 | 89.1 | 83.8 | 89.2 | | -| [IE-SQL (anonymous)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
| 84.6 | 88.7 | 84.6 | 88.8 | | | [(Guo 2019)
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^ | 84.3 | 90.3 | 83.7 | 89.2 | | +| [IE-SQL (Ma 2020)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
(Ping An Life, AI Team) | 84.6 | 88.7 | 84.6 | 88.8 | | | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | | [SQLova
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 81.6 | 87.2 | 80.7 | 86.2 | | | [IncSQL
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 49.9 | 84.0 | 49.9 | 83.7 | | From 397e21d407b66ea72673075a33566fdb6334ebf6 Mon Sep 17 00:00:00 2001 From: Victor Zhong Date: Mon, 17 Aug 2020 18:46:40 -0700 Subject: [PATCH 28/42] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index da55023..75e3c76 100755 --- a/README.md +++ b/README.md @@ -50,12 +50,12 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | -| [Execution-Guided Decoding
(Wang 2018)](https://arxiv.org/abs/1807.03100) | 76.0 | 84.0 | 75.4 | 83.8 | Inference | | [HydraNet (Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 83.6 | 89.1 | 83.8 | 89.2 | | | [(Guo 2019)
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^ | 84.3 | 90.3 | 83.7 | 89.2 | | | [IE-SQL (Ma 2020)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
(Ping An Life, AI Team) | 84.6 | 88.7 | 84.6 | 88.8 | | | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | | [SQLova
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 81.6 | 87.2 | 80.7 | 86.2 | | +| [Execution-Guided Decoding
(Wang 2018)](https://arxiv.org/abs/1807.03100) | 76.0 | 84.0 | 75.4 | 83.8 | Inference | | [IncSQL
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 49.9 | 84.0 | 49.9 | 83.7 | | | [Auxiliary Mapping Task
(Chang 2019)](https://arxiv.org/pdf/1908.11052.pdf) | 76.0 | 82.3 | 75.0 | 81.7 | | | [MQAN (unordered)
(McCann 2018)](https://arxiv.org/abs/1806.08730) | 76.1 | 82.0 | 75.4 | 81.4 | | From a09fea65c8e1b39acd7514dcae78ee7448180b82 Mon Sep 17 00:00:00 2001 From: Victor Zhong Date: Mon, 17 Aug 2020 18:51:16 -0700 Subject: [PATCH 29/42] Update README.md Add BRIDGE result --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 75e3c76..9ada184 100755 --- a/README.md +++ b/README.md @@ -47,10 +47,12 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [IE-SQL
+Execution-Guided Decoding
(Ma 2020)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
(Ping An Life, AI Team)| 87.9 | 92.6 | 87.8 | 92.5 | Inference | | [HydraNet
+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | +| BRIDGE
+Execution-Guided Decoding
(anonymous) | 86.1 | 92.5 | 85.8 | 91.7 | Inference | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | | [HydraNet (Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 83.6 | 89.1 | 83.8 | 89.2 | | +| BRIDGE (anonymous) | 85.1 | 91.1 | 84.8 | 90.4 | | [(Guo 2019)
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^ | 84.3 | 90.3 | 83.7 | 89.2 | | | [IE-SQL (Ma 2020)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
(Ping An Life, AI Team) | 84.6 | 88.7 | 84.6 | 88.8 | | | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | From 61bc17f2b4d97de4c31fb0087b66a8eb7d35a56f Mon Sep 17 00:00:00 2001 From: Victor Zhong Date: Mon, 17 Aug 2020 18:57:02 -0700 Subject: [PATCH 30/42] Update README.md --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 9ada184..4ad7dfa 100755 --- a/README.md +++ b/README.md @@ -24,6 +24,10 @@ If you use WikiSQL, please cite the following work: } ``` +## Notes + +Regarding tokenization and Stanza --- when WikiSQL was written 3-years ago, it relied on Stanza, a CoreNLP python wrapper that has since been deprecated. If you'd still like to use the tokenizer, please use the docker image. We do not anticipate switching to the current Stanza as changes to the tokenizer would render the previous results not reproducible. + ## Leaderboard If you submit papers on WikiSQL, please consider sending a pull request to merge your results onto the leaderboard. By submitting, you acknowledge that your results are obtained purely by training on the training split and tuned on the dev split (e.g. you only evaluted on the test set once). Moreover, you acknowledge that your models only use the table schema and question during inference. That is they do *not* use the table content. **Update (May 12, 2019)**: We now have a separate leaderboard for weakly supervised models that do not use logical forms during training. @@ -120,7 +124,7 @@ These files are contained in the `*.jsonl` files. A line looks like the followin The fields represent the following: -- `phase`: the phase in which the dataset was collection. We collected WikiSQL in two phases. +- `phase`: the phase in which the dataset was collected. We collected WikiSQL in two phases. - `question`: the natural language question written by the worker. - `table_id`: the ID of the table to which this question is addressed. - `sql`: the SQL query corresponding to the question. This has the following subfields: From 9288e2cd9d9c5b0f60a32b96aadecee3e17ecb46 Mon Sep 17 00:00:00 2001 From: Victor Zhong Date: Mon, 17 Aug 2020 18:58:32 -0700 Subject: [PATCH 31/42] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4ad7dfa..0863169 100755 --- a/README.md +++ b/README.md @@ -52,11 +52,11 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [HydraNet
+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | BRIDGE
+Execution-Guided Decoding
(anonymous) | 86.1 | 92.5 | 85.8 | 91.7 | Inference | +| BRIDGE (anonymous) | 85.1 | 91.1 | 84.8 | 90.4 | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | | [HydraNet (Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 83.6 | 89.1 | 83.8 | 89.2 | | -| BRIDGE (anonymous) | 85.1 | 91.1 | 84.8 | 90.4 | | [(Guo 2019)
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^ | 84.3 | 90.3 | 83.7 | 89.2 | | | [IE-SQL (Ma 2020)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
(Ping An Life, AI Team) | 84.6 | 88.7 | 84.6 | 88.8 | | | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | From 793d5a12daf8dff72d8c82b405e994d4da7f0a0d Mon Sep 17 00:00:00 2001 From: Victor Zhong Date: Mon, 17 Aug 2020 18:59:49 -0700 Subject: [PATCH 32/42] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 0863169..4ed5696 100755 --- a/README.md +++ b/README.md @@ -51,8 +51,8 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [IE-SQL
+Execution-Guided Decoding
(Ma 2020)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
(Ping An Life, AI Team)| 87.9 | 92.6 | 87.8 | 92.5 | Inference | | [HydraNet
+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | -| BRIDGE
+Execution-Guided Decoding
(anonymous) | 86.1 | 92.5 | 85.8 | 91.7 | Inference | -| BRIDGE (anonymous) | 85.1 | 91.1 | 84.8 | 90.4 | +| BRIDGE^
+Execution-Guided Decoding
(anonymous) | 86.1 | 92.5 | 85.8 | 91.7 | Inference | +| BRIDGE^ (anonymous) | 85.1 | 91.1 | 84.8 | 90.4 | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | From 65429b6b7194d57491c0396676388b653bce431f Mon Sep 17 00:00:00 2001 From: Louis Li Date: Tue, 10 Nov 2020 02:01:55 +0000 Subject: [PATCH 33/42] updated --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 4ed5696..9c004df 100755 --- a/README.md +++ b/README.md @@ -37,6 +37,8 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev execution accuracy | Test execution accuracy | | :---: | :---: | :---: | +| [SeqGenSQL+EG (Li 2020)](https://arxiv.org/abs/2011.03836) | 90.8 | 90.5 | +| [SeqGenSQL (Li 2020)](https://arxiv.org/abs/2011.03836) | 90.6 | 90.3| | [HardEM (Min 2019)](https://arxiv.org/abs/1909.04849) | 84.4 | 83.9 | | [LatentAlignment (Wang 2019)](https://arxiv.org/abs/1909.04165) | 79.4 | 79.3 | | [MeRL (Agarwal 2019)](https://arxiv.org/abs/1902.07198) | 74.9 +/- 0.1 | 74.8 +/- 0.2 | From 40288704925718bfb73281f5b56a3f793d25e0ce Mon Sep 17 00:00:00 2001 From: Victoria X Lin Date: Wed, 30 Dec 2020 16:37:15 -0800 Subject: [PATCH 34/42] BRIDGE Results Update + ArXiv paper link --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 4ed5696..4a65f09 100755 --- a/README.md +++ b/README.md @@ -50,9 +50,9 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | :---: | :---: | :---: | :---: | :---: | :---: | | [IE-SQL
+Execution-Guided Decoding
(Ma 2020)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
(Ping An Life, AI Team)| 87.9 | 92.6 | 87.8 | 92.5 | Inference | | [HydraNet
+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | +| [BRIDGE^
+Execution-Guided Decoding
(Lin 2020)](https://arxiv.org/abs/2012.12627)
(Salesforce Research) | 86.8 | 92.6 | 86.3 | 91.9 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | -| BRIDGE^
+Execution-Guided Decoding
(anonymous) | 86.1 | 92.5 | 85.8 | 91.7 | Inference | -| BRIDGE^ (anonymous) | 85.1 | 91.1 | 84.8 | 90.4 | +| [BRIDGE^
(Lin 2020)
](https://arxiv.org/abs/2012.12627)(Salesforce Research) | 86.2 | 91.7 | 85.7 | 91.1 | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | From 0dce610b162659e2a0cd3e641063f10889d5bc50 Mon Sep 17 00:00:00 2001 From: Victor Zhong Date: Mon, 8 Mar 2021 17:58:20 -0800 Subject: [PATCH 35/42] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index b59a386..a1f9315 100755 --- a/README.md +++ b/README.md @@ -37,8 +37,6 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev execution accuracy | Test execution accuracy | | :---: | :---: | :---: | -| [SeqGenSQL+EG (Li 2020)](https://arxiv.org/abs/2011.03836) | 90.8 | 90.5 | -| [SeqGenSQL (Li 2020)](https://arxiv.org/abs/2011.03836) | 90.6 | 90.3| | [HardEM (Min 2019)](https://arxiv.org/abs/1909.04849) | 84.4 | 83.9 | | [LatentAlignment (Wang 2019)](https://arxiv.org/abs/1909.04165) | 79.4 | 79.3 | | [MeRL (Agarwal 2019)](https://arxiv.org/abs/1902.07198) | 74.9 +/- 0.1 | 74.8 +/- 0.2 | @@ -55,6 +53,8 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [BRIDGE^
+Execution-Guided Decoding
(Lin 2020)](https://arxiv.org/abs/2012.12627)
(Salesforce Research) | 86.8 | 92.6 | 86.3 | 91.9 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [BRIDGE^
(Lin 2020)
](https://arxiv.org/abs/2012.12627)(Salesforce Research) | 86.2 | 91.7 | 85.7 | 91.1 | +| [SeqGenSQL+EG (Li 2020)](https://arxiv.org/abs/2011.03836) | | 90.8 | | 90.5 | +| [SeqGenSQL (Li 2020)](https://arxiv.org/abs/2011.03836) | | 90.6 | | 90.3| | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | From b73f071fa5559c01ec896b82bcb561637b73d091 Mon Sep 17 00:00:00 2001 From: Victor Zhong Date: Mon, 8 Mar 2021 17:59:01 -0800 Subject: [PATCH 36/42] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index a1f9315..dd0a62b 100755 --- a/README.md +++ b/README.md @@ -53,8 +53,8 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [BRIDGE^
+Execution-Guided Decoding
(Lin 2020)](https://arxiv.org/abs/2012.12627)
(Salesforce Research) | 86.8 | 92.6 | 86.3 | 91.9 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [BRIDGE^
(Lin 2020)
](https://arxiv.org/abs/2012.12627)(Salesforce Research) | 86.2 | 91.7 | 85.7 | 91.1 | -| [SeqGenSQL+EG (Li 2020)](https://arxiv.org/abs/2011.03836) | | 90.8 | | 90.5 | -| [SeqGenSQL (Li 2020)](https://arxiv.org/abs/2011.03836) | | 90.6 | | 90.3| +| [SeqGenSQL+EG (Li 2020)](https://arxiv.org/abs/2011.03836) | | 90.8 | | 90.5 | Inference | +| [SeqGenSQL (Li 2020)](https://arxiv.org/abs/2011.03836) | | 90.6 | | 90.3| | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | From a805103d4c623c001a1ab8007c95cc6de33d5418 Mon Sep 17 00:00:00 2001 From: huybery Date: Tue, 9 Mar 2021 20:32:29 +0800 Subject: [PATCH 37/42] update SDSQL result --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index dd0a62b..8e37c70 100755 --- a/README.md +++ b/README.md @@ -48,10 +48,12 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | +| [SDSQL
+Execution-Guided Decoding
(Hui 2020)](https://arxiv.org/abs/2103.04399)
(Alibaba Group)| 87.1 | 92.6 | 87.0 | 92.7 | Inference | | [IE-SQL
+Execution-Guided Decoding
(Ma 2020)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
(Ping An Life, AI Team)| 87.9 | 92.6 | 87.8 | 92.5 | Inference | | [HydraNet
+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [BRIDGE^
+Execution-Guided Decoding
(Lin 2020)](https://arxiv.org/abs/2012.12627)
(Salesforce Research) | 86.8 | 92.6 | 86.3 | 91.9 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | +| [SDSQL
(Hui 2020)](https://arxiv.org/abs/2103.04399)
(Alibaba Group)| 86.0 | 91.8 | 85.6 | 91.4 | | | [BRIDGE^
(Lin 2020)
](https://arxiv.org/abs/2012.12627)(Salesforce Research) | 86.2 | 91.7 | 85.7 | 91.1 | | [SeqGenSQL+EG (Li 2020)](https://arxiv.org/abs/2011.03836) | | 90.8 | | 90.5 | Inference | | [SeqGenSQL (Li 2020)](https://arxiv.org/abs/2011.03836) | | 90.6 | | 90.3| | From bc0a6a9cf7f6b0159f13d84bb46d71765996c3b7 Mon Sep 17 00:00:00 2001 From: Victor Zhong Date: Tue, 9 Mar 2021 16:58:55 -0800 Subject: [PATCH 38/42] Update README.md --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index dd0a62b..c402287 100755 --- a/README.md +++ b/README.md @@ -53,8 +53,9 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [BRIDGE^
+Execution-Guided Decoding
(Lin 2020)](https://arxiv.org/abs/2012.12627)
(Salesforce Research) | 86.8 | 92.6 | 86.3 | 91.9 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [BRIDGE^
(Lin 2020)
](https://arxiv.org/abs/2012.12627)(Salesforce Research) | 86.2 | 91.7 | 85.7 | 91.1 | +| [Text2SQLGen + EG (Mellah 2021)](https://novelisconsulting-my.sharepoint.com/:b:/g/personal/ymellah_novelis_io/EXQIItFj30BNhD0CytVzIB4BdvpEDLd6CFtq2wNaF5YMvA?e=DHk325)
(Novelis.io Research) | | 91.2 | | 91.0 | Inference | | [SeqGenSQL+EG (Li 2020)](https://arxiv.org/abs/2011.03836) | | 90.8 | | 90.5 | Inference | -| [SeqGenSQL (Li 2020)](https://arxiv.org/abs/2011.03836) | | 90.6 | | 90.3| | +| [SeqGenSQL (Li 2020)](https://arxiv.org/abs/2011.03836) | | 90.6 | | 90.3| Inference | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | From 6fbce67de6ef35025455b68b5389bc0410725b1d Mon Sep 17 00:00:00 2001 From: Qin Lyu Date: Fri, 30 Apr 2021 10:05:44 -0700 Subject: [PATCH 39/42] Add link to HydraNet repo --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index b1a845b..0677124 100755 --- a/README.md +++ b/README.md @@ -50,7 +50,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | :---: | :---: | :---: | :---: | :---: | :---: | | [SDSQL
+Execution-Guided Decoding
(Hui 2020)](https://arxiv.org/abs/2103.04399)
(Alibaba Group)| 87.1 | 92.6 | 87.0 | 92.7 | Inference | | [IE-SQL
+Execution-Guided Decoding
(Ma 2020)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
(Ping An Life, AI Team)| 87.9 | 92.6 | 87.8 | 92.5 | Inference | -| [HydraNet
+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | +| [HydraNet
+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI)
[(code)](https://github.com/lyuqin/HydraNet-WikiSQL)| 86.6 | 92.4 | 86.5 | 92.2 | Inference | | [BRIDGE^
+Execution-Guided Decoding
(Lin 2020)](https://arxiv.org/abs/2012.12627)
(Salesforce Research) | 86.8 | 92.6 | 86.3 | 91.9 | Inference | | [X-SQL
+Execution-Guided Decoding
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 86.2 | 92.3 | 86.0 | 91.8 | Inference | | [SDSQL
(Hui 2020)](https://arxiv.org/abs/2103.04399)
(Alibaba Group)| 86.0 | 91.8 | 85.6 | 91.4 | | @@ -61,7 +61,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | -| [HydraNet (Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 83.6 | 89.1 | 83.8 | 89.2 | | +| [HydraNet (Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI)
[(code)](https://github.com/lyuqin/HydraNet-WikiSQL)| 83.6 | 89.1 | 83.8 | 89.2 | | | [(Guo 2019)
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^ | 84.3 | 90.3 | 83.7 | 89.2 | | | [IE-SQL (Ma 2020)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
(Ping An Life, AI Team) | 84.6 | 88.7 | 84.6 | 88.8 | | | [X-SQL
(He 2019)](https://www.microsoft.com/en-us/research/publication/x-sql-reinforce-context-into-schema-representation/) | 83.8 | 89.5 | 83.3 | 88.7 | | From ee3d66d4c6999f565d0608860026108bfdd5aff4 Mon Sep 17 00:00:00 2001 From: whksmo <715705307@qq.com> Date: Tue, 18 May 2021 19:11:01 +0800 Subject: [PATCH 40/42] Add SeaD results Add SeaD results for WikiSQL benchmark. Arxiv: https://arxiv.org/abs/2105.07911. --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index b1a845b..3c8f50e 100755 --- a/README.md +++ b/README.md @@ -48,6 +48,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev logical form
accuracy | Dev
execution
accuracy | Test
logical form
accuracy | Test
execution
accuracy | Uses execution | | :---: | :---: | :---: | :---: | :---: | :---: | +| [SeaD
+Execution-Guided Decoding
(Xu 2021)](https://arxiv.org/abs/2105.07911)
(Ant Group, Ada & ZhiXiaoBao) | 87.6 | 92.9 | 87.5 | 93.0 | Inference | | [SDSQL
+Execution-Guided Decoding
(Hui 2020)](https://arxiv.org/abs/2103.04399)
(Alibaba Group)| 87.1 | 92.6 | 87.0 | 92.7 | Inference | | [IE-SQL
+Execution-Guided Decoding
(Ma 2020)](https://drive.google.com/file/d/1t3xEltqKpYJGYekAhQ5vYFen1ocHJ3sY/view?usp=sharing)
(Ping An Life, AI Team)| 87.9 | 92.6 | 87.8 | 92.5 | Inference | | [HydraNet
+Execution-Guided Decoding
(Lyu 2020)](https://www.microsoft.com/en-us/research/publication/hybrid-ranking-network-for-text-to-sql/)
(Microsoft Dynamics 365 AI) | 86.6 | 92.4 | 86.5 | 92.2 | Inference | @@ -58,6 +59,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | [Text2SQLGen + EG (Mellah 2021)](https://novelisconsulting-my.sharepoint.com/:b:/g/personal/ymellah_novelis_io/EXQIItFj30BNhD0CytVzIB4BdvpEDLd6CFtq2wNaF5YMvA?e=DHk325)
(Novelis.io Research) | | 91.2 | | 91.0 | Inference | | [SeqGenSQL+EG (Li 2020)](https://arxiv.org/abs/2011.03836) | | 90.8 | | 90.5 | Inference | | [SeqGenSQL (Li 2020)](https://arxiv.org/abs/2011.03836) | | 90.6 | | 90.3| Inference | +| [SeaD
(Xu 2021)](https://arxiv.org/abs/2105.07911)
(Ant Group, Ada & ZhiXiaoBao) | 84.9 | 90.2 | 84.7 | 90.1 | Inference | | [(Guo 2019)
+Execution-Guided Decoding
with BERT-Base-Uncased](https://arxiv.org/abs/1910.07179)^| 85.4 | 91.1 | 84.5 | 90.1 | Inference | | [SQLova
+Execution-Guided Decoding
(Hwang 2019)](https://ssl.pstatic.net/static/clova/service/clova_ai/research/publications/SQLova.pdf) | 84.2 | 90.2 | 83.6 | 89.6 | Inference | | [IncSQL
+Execution-Guided Decoding
(Shi 2018)](https://arxiv.org/pdf/1809.05054.pdf) | 51.3 | 87.2 | 51.1 | 87.1 | Inference | From 0af869a830033f89a026b6c748879e04e1b0e2fc Mon Sep 17 00:00:00 2001 From: svc-scm <48930134+svc-scm@users.noreply.github.com> Date: Tue, 12 Oct 2021 11:50:55 -0700 Subject: [PATCH 41/42] Updated/Added CODEOWNERS with ECCN --- CODEOWNERS | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 CODEOWNERS diff --git a/CODEOWNERS b/CODEOWNERS new file mode 100644 index 0000000..522fa4a --- /dev/null +++ b/CODEOWNERS @@ -0,0 +1,2 @@ +# Comment line immediately above ownership line is reserved for related gus information. Please be careful while editing. +#ECCN:Open Source From e071ac58ca4f1590fbdfda2ace0e7cd97bc542a2 Mon Sep 17 00:00:00 2001 From: Qian Date: Mon, 28 Feb 2022 13:28:13 +0800 Subject: [PATCH 42/42] Add TAPEX experimental results --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index b36cf1d..0106898 100755 --- a/README.md +++ b/README.md @@ -37,6 +37,7 @@ If you submit papers on WikiSQL, please consider sending a pull request to merge | Model | Dev execution accuracy | Test execution accuracy | | :---: | :---: | :---: | +| [TAPEX (Liu 2022)](https://openreview.net/forum?id=O50443AsCP) | 89.2 | 89.5 | | [HardEM (Min 2019)](https://arxiv.org/abs/1909.04849) | 84.4 | 83.9 | | [LatentAlignment (Wang 2019)](https://arxiv.org/abs/1909.04165) | 79.4 | 79.3 | | [MeRL (Agarwal 2019)](https://arxiv.org/abs/1902.07198) | 74.9 +/- 0.1 | 74.8 +/- 0.2 |