From 4fccb5fdaca054905bfab7cb01e7eba6fb47c114 Mon Sep 17 00:00:00 2001 From: chriseth Date: Fri, 10 Jun 2016 17:25:51 +0200 Subject: [PATCH 01/26] Document input description and metadata output. --- docs/index.rst | 1 + docs/miscellaneous.rst | 39 ---------- docs/using-the-compiler.rst | 141 ++++++++++++++++++++++++++++++++++++ 3 files changed, 142 insertions(+), 39 deletions(-) create mode 100644 docs/using-the-compiler.rst diff --git a/docs/index.rst b/docs/index.rst index cb79687b9..904c3a544 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -130,6 +130,7 @@ Contents solidity-by-example.rst solidity-in-depth.rst security-considerations.rst + using-the-compiler.rst style-guide.rst common-patterns.rst contributing.rst diff --git a/docs/miscellaneous.rst b/docs/miscellaneous.rst index cc40d6a75..476500676 100644 --- a/docs/miscellaneous.rst +++ b/docs/miscellaneous.rst @@ -221,45 +221,6 @@ This means the following source mappings represent the same information: ``1:2:1;:9;2::2;;`` - -.. index:: ! commandline compiler, compiler;commandline, ! solc, ! linker - -.. _commandline-compiler: - -****************************** -Using the Commandline Compiler -****************************** - -One of the build targets of the Solidity repository is ``solc``, the solidity commandline compiler. -Using ``solc --help`` provides you with an explanation of all options. The compiler can produce various outputs, ranging from simple binaries and assembly over an abstract syntax tree (parse tree) to estimations of gas usage. -If you only want to compile a single file, you run it as ``solc --bin sourceFile.sol`` and it will print the binary. Before you deploy your contract, activate the optimizer while compiling using ``solc --optimize --bin sourceFile.sol``. If you want to get some of the more advanced output variants of ``solc``, it is probably better to tell it to output everything to separate files using ``solc -o outputDirectory --bin --ast --asm sourceFile.sol``. - -The commandline compiler will automatically read imported files from the filesystem, but -it is also possible to provide path redirects using ``context:prefix=path`` in the following way: - -:: - - solc github.com/ethereum/dapp-bin/=/usr/local/lib/dapp-bin/ =/usr/local/lib/fallback file.sol - -This essentially instructs the compiler to search for anything starting with -``github.com/ethereum/dapp-bin/`` under ``/usr/local/lib/dapp-bin`` and if it does not -find the file there, it will look at ``/usr/local/lib/fallback`` (the empty prefix -always matches). ``solc`` will not read files from the filesystem that lie outside of -the remapping targets and outside of the directories where explicitly specified source -files reside, so things like ``import "/etc/passwd";`` only work if you add ``=/`` as a remapping. - -You can restrict remappings to only certain source files by prefixing a context. - -The section on :ref:`import` provides more details on remappings. - -If there are multiple matches due to remappings, the one with the longest common prefix is selected. - -If your contracts use :ref:`libraries `, you will notice that the bytecode contains substrings of the form ``__LibraryName______``. You can use ``solc`` as a linker meaning that it will insert the library addresses for you at those points: - -Either add ``--libraries "Math:0x12345678901234567890 Heap:0xabcdef0123456"`` to your command to provide an address for each library or store the string in a file (one library per line) and run ``solc`` using ``--libraries fileName``. - -If ``solc`` is called with the option ``--link``, all input files are interpreted to be unlinked binaries (hex-encoded) in the ``__LibraryName____``-format given above and are linked in-place (if the input is read from stdin, it is written to stdout). All options except ``--libraries`` are ignored (including ``-o``) in this case. - ***************** Contract Metadata ***************** diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst new file mode 100644 index 000000000..47d7c7175 --- /dev/null +++ b/docs/using-the-compiler.rst @@ -0,0 +1,141 @@ +.. index:: ! commandline compiler, compiler;commandline, ! solc, ! linker + +.. _commandline-compiler: + +****************************** +Using the Commandline Compiler +****************************** + +One of the build targets of the Solidity repository is ``solc``, the solidity commandline compiler. +Using ``solc --help`` provides you with an explanation of all options. The compiler can produce various outputs, ranging from simple binaries and assembly over an abstract syntax tree (parse tree) to estimations of gas usage. +If you only want to compile a single file, you run it as ``solc --bin sourceFile.sol`` and it will print the binary. Before you deploy your contract, activate the optimizer while compiling using ``solc --optimize --bin sourceFile.sol``. If you want to get some of the more advanced output variants of ``solc``, it is probably better to tell it to output everything to separate files using ``solc -o outputDirectory --bin --ast --asm sourceFile.sol``. + +The commandline compiler will automatically read imported files from the filesystem, but +it is also possible to provide path redirects using ``prefix=path`` in the following way: + +:: + + solc github.com/ethereum/dapp-bin/=/usr/local/lib/dapp-bin/ =/usr/local/lib/fallback file.sol + +This essentially instructs the compiler to search for anything starting with +``github.com/ethereum/dapp-bin/`` under ``/usr/local/lib/dapp-bin`` and if it does not +find the file there, it will look at ``/usr/local/lib/fallback`` (the empty prefix +always matches). ``solc`` will not read files from the filesystem that lie outside of +the remapping targets and outside of the directories where explicitly specified source +files reside, so things like ``import "/etc/passwd";`` only work if you add ``=/`` as a remapping. + +If there are multiple matches due to remappings, the one with the longest common prefix is selected. + +If your contracts use :ref:`libraries `, you will notice that the bytecode contains substrings of the form ``__LibraryName______``. You can use ``solc`` as a linker meaning that it will insert the library addresses for you at those points: + +Either add ``--libraries "Math:0x12345678901234567890 Heap:0xabcdef0123456"`` to your command to provide an address for each library or store the string in a file (one library per line) and run ``solc`` using ``--libraries fileName``. + +If ``solc`` is called with the option ``--link``, all input files are interpreted to be unlinked binaries (hex-encoded) in the ``__LibraryName____``-format given above and are linked in-place (if the input is read from stdin, it is written to stdout). All options except ``--libraries`` are ignored (including ``-o``) in this case. + + +************************************************** +Standardized Input Description and Metadata Output +************************************************** + +In order to ease source code verification of complex contracts that are spread across several files, +there is a standardized for describing the relations between those files. +Furthermore, the compiler can generate a json file while compiling that includes +the source, natspec comments and other metadata whose hash is included in the +actual bytecode. + +There is some overlap between the input description and the metadata output +and due to the fact that some fields are optional, the metadata can be used as +input to the compiler. In order to verify the metadata, you actually take it, +re-run the compiler on the metadata and check that it again produces the same +metadata. + +If the compiler is invoked in a different way, not using the input +description (for example by using a file content retrieval callback), +the compiler can still generate the metadata alongside the bytecode of each +contract. + +The metadata standard is versioned. Future versions are only required to provide the "version" field, +the two keys inside the "compiler" field. The field compiler.keccak should be the keccak hash +of a binary of the compiler with the given version. + +The example below is presented in a human-readable way. Properly formatted metadata +should use quotes correctly, reduce whitespace to a minimum and sort the keys of all objects +to arrive at a unique formatting. + +Comments are of course not permitted and used here only for explanatory purposes. + +Input Description +----------------- + +The input description could change with each compiler version, but it +should be backwards compatible if possible. + + { + sources: + { + "abc": "contract b{}", + "def": {keccak: "0x123..."}, // source has to be retrieved by its hash + "dir/file.sol": "contract a {}" + }, + settings: + { + remappings: [":g/dir"], + optimizer: {enabled: true, runs: 500}, + compilationTarget: "myFile.sol:MyContract", // Can also be an array + // To be backwards compatible, use the full name of the contract in the output + // only if there are name clashes. + // If the full name was given as "compilationTargets", use the full name. + libraries: { + "def:MyLib": "0x123123..." + }, + // The following can be used to restrict the fields the compiler will output. + outputSelection: { + // to be defined + } + } + } + +Metadata Output +--------------- + +Note that the actual bytecode is not part of the metadata because the hash +of the metadata structure will be included in the bytecode itself. + +This requires the compiler to be able to compute the hash of its own binary, +which requires it to be statically linked. The hash of the binary is not +too important. It is much more important to have the commit hash because +that can be used to query a location of the binary (and whether the version is +"official") at a registry contract. + + { + version: "1", + compiler: { + version: "soljson-2313-2016-12-12", + keccak: "0x123..." + }, + sources: + { + "abc": "contract b{}", + "def": {keccak: "0x123..."}, // source has to be retrieved by its hash + "dir/file.sol": "contract a {}" + }, + settings: + { + remappings: [":g/dir"], + optimizer: {enabled: true, runs: 500}, + compilationTarget: "myFile.sol:MyContract", + // To be backwards compatible, use the full name of the contract in the output + // only if there are name clashes. + // If the full name was given as "compilationTargets", use the full name. + libraries: { + "def:MyLib": "0x123123..." + } + }, + output: + { + abi: [ /* abi definition */ ], + userDocumentation: [ /* user documentation comments */ ], + developerDocumentation: [ /* developer documentation comments */ ], + natspec: [ /* natspec comments */ ] + } + } From 57662e1bf3cba937289c50b3b711918a456101a6 Mon Sep 17 00:00:00 2001 From: chriseth Date: Fri, 10 Jun 2016 22:03:22 +0200 Subject: [PATCH 02/26] Add language and some minor corrections and clarifications. --- docs/using-the-compiler.rst | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 47d7c7175..8ceb52a74 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -38,10 +38,11 @@ Standardized Input Description and Metadata Output ************************************************** In order to ease source code verification of complex contracts that are spread across several files, -there is a standardized for describing the relations between those files. +there is a standardized way for describing the relations between those files. Furthermore, the compiler can generate a json file while compiling that includes the source, natspec comments and other metadata whose hash is included in the -actual bytecode. +actual bytecode. Specifically, the creation data for a contract has to begin with +`push32 pop`. There is some overlap between the input description and the metadata output and due to the fact that some fields are optional, the metadata can be used as @@ -55,8 +56,8 @@ the compiler can still generate the metadata alongside the bytecode of each contract. The metadata standard is versioned. Future versions are only required to provide the "version" field, -the two keys inside the "compiler" field. The field compiler.keccak should be the keccak hash -of a binary of the compiler with the given version. +the "language" field and the two keys inside the "compiler" field. +The field compiler.keccak should be the keccak hash of a binary of the compiler with the given version. The example below is presented in a human-readable way. Properly formatted metadata should use quotes correctly, reduce whitespace to a minimum and sort the keys of all objects @@ -67,7 +68,7 @@ Comments are of course not permitted and used here only for explanatory purposes Input Description ----------------- -The input description could change with each compiler version, but it +The input description is language-specific and could change with each compiler version, but it should be backwards compatible if possible. { @@ -105,11 +106,13 @@ This requires the compiler to be able to compute the hash of its own binary, which requires it to be statically linked. The hash of the binary is not too important. It is much more important to have the commit hash because that can be used to query a location of the binary (and whether the version is -"official") at a registry contract. +"official") at a registry contract. { version: "1", + language: "Solidity", compiler: { + commit: "55db20e32c97098d13230ab7500758e8e3b31d64", version: "soljson-2313-2016-12-12", keccak: "0x123..." }, From 77b934c86118fe9148265dbb6b4db9ed1335ac27 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Wed, 16 Nov 2016 13:05:24 +0000 Subject: [PATCH 03/26] Update with https://pad.riseup.net/p/7x3G896a3NLA --- docs/using-the-compiler.rst | 148 +++++++++++++++++++++++++++++------- 1 file changed, 121 insertions(+), 27 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 8ceb52a74..5b8a4b5c2 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -40,21 +40,10 @@ Standardized Input Description and Metadata Output In order to ease source code verification of complex contracts that are spread across several files, there is a standardized way for describing the relations between those files. Furthermore, the compiler can generate a json file while compiling that includes -the source, natspec comments and other metadata whose hash is included in the +the (hash of the) source, natspec comments and other metadata whose hash is included in the actual bytecode. Specifically, the creation data for a contract has to begin with `push32 pop`. -There is some overlap between the input description and the metadata output -and due to the fact that some fields are optional, the metadata can be used as -input to the compiler. In order to verify the metadata, you actually take it, -re-run the compiler on the metadata and check that it again produces the same -metadata. - -If the compiler is invoked in a different way, not using the input -description (for example by using a file content retrieval callback), -the compiler can still generate the metadata alongside the bytecode of each -contract. - The metadata standard is versioned. Future versions are only required to provide the "version" field, the "language" field and the two keys inside the "compiler" field. The field compiler.keccak should be the keccak hash of a binary of the compiler with the given version. @@ -68,34 +57,128 @@ Comments are of course not permitted and used here only for explanatory purposes Input Description ----------------- +QUESTION: How to specific file-reading callback? - probably not as part of json input + The input description is language-specific and could change with each compiler version, but it should be backwards compatible if possible. { sources: { - "abc": "contract b{}", + // the keys here are the "global" names of the source files, imports can use other files via remappings (see below) + "abc": "contract b{}", // specify source directly + // (axic) I think 'keccak' on its on is not enough. I would go perhaps with swarm: "0x12.." and ipfs: "Qma..." for simplicity + // (chriseth) Where the content is stored is a second component, but yes, we could give an indication there. "def": {keccak: "0x123..."}, // source has to be retrieved by its hash + "ghi": {file: "/tmp/path/to/file.sol"}, // file on filesystem + // (axic) I'm inclined to think the source _must_ be provided in the JSON, + "dir/file.sol": "contract a {}" }, settings: { - remappings: [":g/dir"], + remappings: [":g/dir"], // just as it used to be + // (axic) what is remapping doing exactly? optimizer: {enabled: true, runs: 500}, - compilationTarget: "myFile.sol:MyContract", // Can also be an array - // To be backwards compatible, use the full name of the contract in the output - // only if there are name clashes. - // If the full name was given as "compilationTargets", use the full name. + // if given, only compiles this contract, can also be an array. If only a contract name is given, tries to find it if unique. + compilationTarget: "myFile.sol:MyContract", + // addresses of the libraries. If not all libraries are given here, it can result in unlinked objects whose output data is different libraries: { "def:MyLib": "0x123123..." }, // The following can be used to restrict the fields the compiler will output. + // (axic) + outputSelection: [ + "abi", "evm.assembly", "evm.bytecode", ..., "why3", "ewasm.wasm" + ] outputSelection: { + abi,asm,ast,bin,bin-runtime,clone-bin,devdoc,interface,opcodes,srcmap,srcmap-runtime,userdoc + + --ast AST of all source files. + --ast-json AST of all source files in JSON format. + --asm EVM assembly of the contracts. + --asm-json EVM assembly of the contracts in JSON format. + --opcodes Opcodes of the contracts. + --bin Binary of the contracts in hex. + --bin-runtime Binary of the runtime part of the contracts in hex. + --clone-bin Binary of the clone contracts in hex. + --abi ABI specification of the contracts. + --interface Solidity interface of the contracts. + --hashes Function signature hashes of the contracts. + --userdoc Natspec user documentation of all contracts. + --devdoc Natspec developer documentation of all contracts. + --formal Translated source suitable for formal analysis. + // to be defined } } } + +Regular Output +-------------- + + + { + errors: ["error1", "error2"], // we might structure them + errors: [ + { + // (axic) + file: "sourceFile.sol", // optional? + contract: "contractName", // optional + line: 100, // optional - currently, we always have a byte range in the source file + // Errors/warnings originate in several components, most of them are not + // backend-specific. Currently, why3 errors are part of the why3 output. + // I think it is better to put code-generator-specific errors into the code-generator output + // area, and warnings and errors that are code-generator-agnostic into this general area, + // so that it is easier to determine whether some source code is invalid or only + // triggers errors/warnings in some backend that might only implement some part of solidity. + type: "evm" or "why3" or "ewasm" // maybe a better field name would be needed + severity: "warning" or "error" // mandatory + message: "Invalid keyword" // mandatory + } + ] + contracts: { + "sourceFile.sol:ContractName": { + abi: + evm: { + assembly: + bytecode: + runtimeBytecode: + opcodes: + annotatedOpcodes: // (axic) see https://github.com/ethereum/solidity/issues/1178 + gasEstimates: + sourceMap: + runtimeSourceMap: + // If given, this is an unlinked object (cannot be filtered out explicitly, might be + // filtered if both bytecode, runtimeBytecode, opcodes and others are filtered out) + linkReferences: { + "sourceFile.sol:Library1": [1, 200, 80] // byte offsets into bytecode. Linking replaces the 20 bytes there. + } + // the same for runtimeBytecode - I'm not sure it is a good idea to allow to link libraries differently for the runtime bytecode. + // furthermore, runtime bytecode is always a substring of the bytecode anyway. + runtimeLinkReferences: { + } + }, + functionHashes: + metadata: // see below + ewasm: { + wast: // S-expression format + wasm: // + } + } + }, + formal: { + "why3": "..." + }, + sourceList: ["source1.sol", "source2.sol"], // this is important for source references both in the ast as well as in the srcmap in the contract + sources: { + "source1.sol": { + "AST": { ... } + } + } + } + Metadata Output --------------- @@ -118,18 +201,15 @@ that can be used to query a location of the binary (and whether the version is }, sources: { - "abc": "contract b{}", - "def": {keccak: "0x123..."}, // source has to be retrieved by its hash - "dir/file.sol": "contract a {}" + "abc": {keccak: "0x456..."}, // here, sources are always given by hash + "def": {keccak: "0x123..."}, + "dir/file.sol": {keccax: "0xabc..."} }, settings: { remappings: [":g/dir"], optimizer: {enabled: true, runs: 500}, compilationTarget: "myFile.sol:MyContract", - // To be backwards compatible, use the full name of the contract in the output - // only if there are name clashes. - // If the full name was given as "compilationTargets", use the full name. libraries: { "def:MyLib": "0x123123..." } @@ -137,8 +217,22 @@ that can be used to query a location of the binary (and whether the version is output: { abi: [ /* abi definition */ ], - userDocumentation: [ /* user documentation comments */ ], - developerDocumentation: [ /* developer documentation comments */ ], - natspec: [ /* natspec comments */ ] + natspec: [ /* user documentation comments */ ] } } + +This is used in the following way: A component that wants to interact +with a contract (e.g. mist) retrieves the creation transaction of the contract +and from that the first 33 bytes. If the first byte decodes into a PUSH32 +instruction, the other 32 bytes are interpreted as the keccak-hash of +a file which is retrieved via a content-addressable storage like swarm. +That file is JSON-decoded into a structure like above. Sources are +retrieved in the same way and combined with the structure into a proper +compiler input description, which selects only the bytecode as output. + +The compiler of the correct version (which is checked to be part of the "official" compilers) +is invoked on that input. The resulting +bytecode is compared (excess bytecode in the creation transaction +is constructor input data) which automatically verifies the metadata since +its hash is part of the bytecode. The constructor input data is decoded +according to the interface and presented to the user. From 0b3f1a5378daa4075820a6be94afb2d9106c3ebd Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Wed, 16 Nov 2016 13:20:10 +0000 Subject: [PATCH 04/26] Describe the ABI output field --- docs/using-the-compiler.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 5b8a4b5c2..72166f181 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -140,7 +140,9 @@ Regular Output ] contracts: { "sourceFile.sol:ContractName": { - abi: + // The Ethereum Contract ABI. If empty, it is represented as an empty array. + // See https://github.com/ethereum/wiki/wiki/Ethereum-Contract-ABI + abi: [], evm: { assembly: bytecode: From 04089edc4e651493f7596bf2372c7c0487fc2dad Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Wed, 16 Nov 2016 13:20:34 +0000 Subject: [PATCH 05/26] Add missing fields --- docs/using-the-compiler.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 72166f181..a49bfb8e5 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -167,7 +167,10 @@ Regular Output ewasm: { wast: // S-expression format wasm: // - } + }, + userdoc: // Obsolete + devdoc: // Obsolete + natspec: // Combined dev+userdoc } }, formal: { From 073871c248b097b654fb354fd756559009d7c3e9 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Wed, 16 Nov 2016 13:25:40 +0000 Subject: [PATCH 06/26] Update the metadata JSON spec --- docs/using-the-compiler.rst | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index a49bfb8e5..4fc5c71d1 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -204,12 +204,15 @@ that can be used to query a location of the binary (and whether the version is version: "soljson-2313-2016-12-12", keccak: "0x123..." }, + // This is a subset of the regular compiler input sources: { "abc": {keccak: "0x456..."}, // here, sources are always given by hash "def": {keccak: "0x123..."}, - "dir/file.sol": {keccax: "0xabc..."} + "dir/file.sol": {keccax: "0xabc..."}, + "xkcd": {swarm: "0x456..."} }, + // This is a subset of the regular compiler input settings: { remappings: [":g/dir"], @@ -219,9 +222,12 @@ that can be used to query a location of the binary (and whether the version is "def:MyLib": "0x123123..." } }, + // This is a subset of the regular compiler output output: { abi: [ /* abi definition */ ], + userdoc: [], + devdoc: [], natspec: [ /* user documentation comments */ ] } } From 559c4c7a451bbe0518441bd0442b8325f40b279a Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 17 Nov 2016 10:31:46 +0000 Subject: [PATCH 07/26] Update the metadata JSON spec --- docs/using-the-compiler.rst | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 4fc5c71d1..de9e104ac 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -197,29 +197,43 @@ that can be used to query a location of the binary (and whether the version is "official") at a registry contract. { + // The version of the metadata format (required field) version: "1", + // Required field language: "Solidity", + // Required field, the contents are specific to the language compiler: { - commit: "55db20e32c97098d13230ab7500758e8e3b31d64", - version: "soljson-2313-2016-12-12", - keccak: "0x123..." + name: "solc", + version: "0.4.5-nightly.2016.11.15+commit.c1b1efaf.Emscripten.clang", + // Optional hash of the compiler binary which produced this output + keccak256: "0x123..." }, // This is a subset of the regular compiler input sources: { - "abc": {keccak: "0x456..."}, // here, sources are always given by hash - "def": {keccak: "0x123..."}, - "dir/file.sol": {keccax: "0xabc..."}, - "xkcd": {swarm: "0x456..."} + "myFile.sol": { + "keccak256": "0x123...", + "url": "bzzr://0x56..." + }, + "Token": { + "keccak256": "0x456...", + "url": "https://raw.githubusercontent.com/ethereum/solidity/develop/std/Token.sol" + }, + "mortal": { + "content": "contract mortal is owned { function kill() { if (msg.sender == owner) selfdestruct(owner); } }" + } }, // This is a subset of the regular compiler input + // Its content is specific to the compiler (determined by the language and compiler fields) settings: { remappings: [":g/dir"], optimizer: {enabled: true, runs: 500}, - compilationTarget: "myFile.sol:MyContract", + compilationTarget: { + "myFile.sol": MyContract" + }, libraries: { - "def:MyLib": "0x123123..." + "MyLib": "0x123123..." } }, // This is a subset of the regular compiler output @@ -228,7 +242,6 @@ that can be used to query a location of the binary (and whether the version is abi: [ /* abi definition */ ], userdoc: [], devdoc: [], - natspec: [ /* user documentation comments */ ] } } From d9f14e77371b09b419e6561cf847cd4d0e72bde0 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Fri, 2 Dec 2016 10:36:11 +0000 Subject: [PATCH 08/26] The metadata section has been moved, make only a reference to it --- docs/using-the-compiler.rst | 100 ++---------------------------------- 1 file changed, 5 insertions(+), 95 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index de9e104ac..0fec6c8c2 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -33,24 +33,11 @@ Either add ``--libraries "Math:0x12345678901234567890 Heap:0xabcdef0123456"`` to If ``solc`` is called with the option ``--link``, all input files are interpreted to be unlinked binaries (hex-encoded) in the ``__LibraryName____``-format given above and are linked in-place (if the input is read from stdin, it is written to stdout). All options except ``--libraries`` are ignored (including ``-o``) in this case. -************************************************** -Standardized Input Description and Metadata Output -************************************************** +***************************************** +Standardized Input and Output Description +***************************************** -In order to ease source code verification of complex contracts that are spread across several files, -there is a standardized way for describing the relations between those files. -Furthermore, the compiler can generate a json file while compiling that includes -the (hash of the) source, natspec comments and other metadata whose hash is included in the -actual bytecode. Specifically, the creation data for a contract has to begin with -`push32 pop`. - -The metadata standard is versioned. Future versions are only required to provide the "version" field, -the "language" field and the two keys inside the "compiler" field. -The field compiler.keccak should be the keccak hash of a binary of the compiler with the given version. - -The example below is presented in a human-readable way. Properly formatted metadata -should use quotes correctly, reduce whitespace to a minimum and sort the keys of all objects -to arrive at a unique formatting. +The compiler API expects a JSON formatted input and outputs the compilations result in a JSON formatted output. Comments are of course not permitted and used here only for explanatory purposes. @@ -163,7 +150,7 @@ Regular Output } }, functionHashes: - metadata: // see below + metadata: // see the Metadata Output documentation ewasm: { wast: // S-expression format wasm: // @@ -183,80 +170,3 @@ Regular Output } } } - -Metadata Output ---------------- - -Note that the actual bytecode is not part of the metadata because the hash -of the metadata structure will be included in the bytecode itself. - -This requires the compiler to be able to compute the hash of its own binary, -which requires it to be statically linked. The hash of the binary is not -too important. It is much more important to have the commit hash because -that can be used to query a location of the binary (and whether the version is -"official") at a registry contract. - - { - // The version of the metadata format (required field) - version: "1", - // Required field - language: "Solidity", - // Required field, the contents are specific to the language - compiler: { - name: "solc", - version: "0.4.5-nightly.2016.11.15+commit.c1b1efaf.Emscripten.clang", - // Optional hash of the compiler binary which produced this output - keccak256: "0x123..." - }, - // This is a subset of the regular compiler input - sources: - { - "myFile.sol": { - "keccak256": "0x123...", - "url": "bzzr://0x56..." - }, - "Token": { - "keccak256": "0x456...", - "url": "https://raw.githubusercontent.com/ethereum/solidity/develop/std/Token.sol" - }, - "mortal": { - "content": "contract mortal is owned { function kill() { if (msg.sender == owner) selfdestruct(owner); } }" - } - }, - // This is a subset of the regular compiler input - // Its content is specific to the compiler (determined by the language and compiler fields) - settings: - { - remappings: [":g/dir"], - optimizer: {enabled: true, runs: 500}, - compilationTarget: { - "myFile.sol": MyContract" - }, - libraries: { - "MyLib": "0x123123..." - } - }, - // This is a subset of the regular compiler output - output: - { - abi: [ /* abi definition */ ], - userdoc: [], - devdoc: [], - } - } - -This is used in the following way: A component that wants to interact -with a contract (e.g. mist) retrieves the creation transaction of the contract -and from that the first 33 bytes. If the first byte decodes into a PUSH32 -instruction, the other 32 bytes are interpreted as the keccak-hash of -a file which is retrieved via a content-addressable storage like swarm. -That file is JSON-decoded into a structure like above. Sources are -retrieved in the same way and combined with the structure into a proper -compiler input description, which selects only the bytecode as output. - -The compiler of the correct version (which is checked to be part of the "official" compilers) -is invoked on that input. The resulting -bytecode is compared (excess bytecode in the creation transaction -is constructor input data) which automatically verifies the metadata since -its hash is part of the bytecode. The constructor input data is decoded -according to the interface and presented to the user. From 720cf20855b7695f3c2059c2e1c52e2cb2501995 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Fri, 2 Dec 2016 13:46:56 +0000 Subject: [PATCH 09/26] Place into a code block --- docs/using-the-compiler.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 0fec6c8c2..dcd5e5892 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -49,6 +49,8 @@ QUESTION: How to specific file-reading callback? - probably not as part of json The input description is language-specific and could change with each compiler version, but it should be backwards compatible if possible. +.. code-block:: none + { sources: { @@ -105,6 +107,7 @@ should be backwards compatible if possible. Regular Output -------------- +.. code-block:: none { errors: ["error1", "error2"], // we might structure them From 6e2cc081ecec5c5c20945b70159d36b9e8286d0a Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 2 Feb 2017 13:54:21 +0000 Subject: [PATCH 10/26] Update sources definition based on the metadata --- docs/using-the-compiler.rst | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index dcd5e5892..57135df88 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -52,17 +52,33 @@ should be backwards compatible if possible. .. code-block:: none { + // Required sources: { - // the keys here are the "global" names of the source files, imports can use other files via remappings (see below) - "abc": "contract b{}", // specify source directly - // (axic) I think 'keccak' on its on is not enough. I would go perhaps with swarm: "0x12.." and ipfs: "Qma..." for simplicity - // (chriseth) Where the content is stored is a second component, but yes, we could give an indication there. - "def": {keccak: "0x123..."}, // source has to be retrieved by its hash - "ghi": {file: "/tmp/path/to/file.sol"}, // file on filesystem - // (axic) I'm inclined to think the source _must_ be provided in the JSON, - - "dir/file.sol": "contract a {}" + // The keys here are the "global" names of the source files, + // imports can use other files via remappings (see below). + "myFile.sol": + { + // Optional: keccak256 hash of the source file + "keccak256": "0x123...", + // Required (unless "content" is used, see below): URL(s) to the source file. + // URL(s) should be imported in this order and the result checked against the + // keccak256 hash (if available). If the hash doesn't match or none of the + // URL(s) result in success, an error should be raised. + "urls": + [ + "bzzr://56ab...", + "ipfs://Qma...", + "file:///tmp/path/to/file.sol" + ] + }, + "mortal": + { + // Optional: keccak256 hash of the source file + "keccak256": "0x234...", + // Required (unless "urls" is used): literal contents of the source file + "content": "contract mortal is owned { function kill() { if (msg.sender == owner) selfdestruct(owner); } }" + } }, settings: { From 82c0e4de1df0d4b17e01e7fd35ebc312156b85fd Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 2 Feb 2017 13:58:30 +0000 Subject: [PATCH 11/26] Update settings section --- docs/using-the-compiler.rst | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 57135df88..2238a773b 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -80,17 +80,29 @@ should be backwards compatible if possible. "content": "contract mortal is owned { function kill() { if (msg.sender == owner) selfdestruct(owner); } }" } }, + // Optional settings: { - remappings: [":g/dir"], // just as it used to be - // (axic) what is remapping doing exactly? - optimizer: {enabled: true, runs: 500}, - // if given, only compiles this contract, can also be an array. If only a contract name is given, tries to find it if unique. - compilationTarget: "myFile.sol:MyContract", - // addresses of the libraries. If not all libraries are given here, it can result in unlinked objects whose output data is different - libraries: { - "def:MyLib": "0x123123..." + // Optional: Sorted list of remappings + remappings: [ ":g/dir" ], + // Optional: Optimizer settings (enabled defaults to false) + optimizer: { + enabled: true, + runs: 500 }, + // If given, only compiles the specified contracts. + compilationTarget: { + "myFile.sol": "MyContract" + }, + // Addresses of the libraries. If not all libraries are given here, it can result in unlinked objects whose output data is different. + libraries: { + // The top level key is the the name of the source file where the library is used. + // If remappings are used, this source file should match the global path after remappings were applied. + // If this key is an empty string, that refers to a global level. + "myFile.sol": { + "MyLib": "0x123123..." + } + } // The following can be used to restrict the fields the compiler will output. // (axic) outputSelection: [ @@ -144,6 +156,7 @@ Regular Output message: "Invalid keyword" // mandatory } ] + // This contains all the compiled outputs. It can be limited/filtered by the compilationTarget setting. contracts: { "sourceFile.sol:ContractName": { // The Ethereum Contract ABI. If empty, it is represented as an empty array. From 4b5639bf63f2c86604cb55387324e00759a0b1f2 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 2 Feb 2017 14:57:07 +0000 Subject: [PATCH 12/26] Update output selection --- docs/using-the-compiler.rst | 67 ++++++++++++++++++++++--------------- 1 file changed, 40 insertions(+), 27 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 2238a773b..ebb03832a 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -90,10 +90,6 @@ should be backwards compatible if possible. enabled: true, runs: 500 }, - // If given, only compiles the specified contracts. - compilationTarget: { - "myFile.sol": "MyContract" - }, // Addresses of the libraries. If not all libraries are given here, it can result in unlinked objects whose output data is different. libraries: { // The top level key is the the name of the source file where the library is used. @@ -103,30 +99,47 @@ should be backwards compatible if possible. "MyLib": "0x123123..." } } - // The following can be used to restrict the fields the compiler will output. - // (axic) - outputSelection: [ - "abi", "evm.assembly", "evm.bytecode", ..., "why3", "ewasm.wasm" - ] + // The following can be used to select desired outputs. + // If this field is omitted, then the compiler loads and does type checking, but will not generate any outputs apart from errors. + // The first level key is the file name and the second is the contract name, where empty contract name refers to the file itself, + // while the star refers to all of the contracts. + // + // The available output types are as follows: + // abi - ABI + // ast - AST of all source files + // why3 - Why3 translated output + // devdoc - Developer documentation (natspec) + // userdoc - User documentation (natspec) + // metadata - Metadata + // evm.ir - New assembly format before desugaring (not supported atm) + // evm.assembly - New assembly format after desugaring (not supported atm) + // evm.asm - Current assembly format (--asm) + // evm.asmJSON - Current assembly format in JSON (--asm-json) + // evm.opcodes - Opcodes list + // evm.methodIdentifiers - The list of function hashes + // evm.gasEstimates - Function gas estimates + // evm.bytecode - Bytecode (--bin) + // evm.deployedBytecode - Deployed bytecode (--bin-runtime) + // evm.sourceMap - Source mapping (useful for debugging) + // ewasm.wast - eWASM S-expressions format (not supported atm) + // ewasm.wasm - eWASM binary format (not supported atm) outputSelection: { - abi,asm,ast,bin,bin-runtime,clone-bin,devdoc,interface,opcodes,srcmap,srcmap-runtime,userdoc - - --ast AST of all source files. - --ast-json AST of all source files in JSON format. - --asm EVM assembly of the contracts. - --asm-json EVM assembly of the contracts in JSON format. - --opcodes Opcodes of the contracts. - --bin Binary of the contracts in hex. - --bin-runtime Binary of the runtime part of the contracts in hex. - --clone-bin Binary of the clone contracts in hex. - --abi ABI specification of the contracts. - --interface Solidity interface of the contracts. - --hashes Function signature hashes of the contracts. - --userdoc Natspec user documentation of all contracts. - --devdoc Natspec developer documentation of all contracts. - --formal Translated source suitable for formal analysis. - - // to be defined + // Enable the metadata and bytecode outputs of every single contract. + "*": { + "*": [ "metadata", "evm.bytecode" ] + }, + // Enable the abi and opcodes output of MyContract defined in file def. + "def": { + "MyContract": [ "abi", "evm.opcodes" ] + }, + // Enable the source map output of every single contract. + "*": { + "*": [ "evm.sourceMap" ] + }, + // Enable the AST and Why3 output of every single file. + "*": { + "": [ "ast", "why3" ] + } } } } From cbb668672fe14f30ae51a022e48f27ddcbe9b324 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 2 Feb 2017 15:00:01 +0000 Subject: [PATCH 13/26] Add metadata.useLiteralContent option --- docs/using-the-compiler.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index ebb03832a..2518310b6 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -90,6 +90,11 @@ should be backwards compatible if possible. enabled: true, runs: 500 }, + // Metadata settings (optional) + metadata: { + // Use only literal content and not URLs (false by default) + useLiteralContent: true + }, // Addresses of the libraries. If not all libraries are given here, it can result in unlinked objects whose output data is different. libraries: { // The top level key is the the name of the source file where the library is used. From d46ec20f88feac1326381e5ac67a02086c5f4516 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 2 Feb 2017 20:52:59 +0000 Subject: [PATCH 14/26] Change layout and include API, Input, Output sections --- docs/using-the-compiler.rst | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 2518310b6..b89af912d 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -1,8 +1,11 @@ +****************** +Using the compiler +****************** + .. index:: ! commandline compiler, compiler;commandline, ! solc, ! linker .. _commandline-compiler: -****************************** Using the Commandline Compiler ****************************** @@ -32,23 +35,26 @@ Either add ``--libraries "Math:0x12345678901234567890 Heap:0xabcdef0123456"`` to If ``solc`` is called with the option ``--link``, all input files are interpreted to be unlinked binaries (hex-encoded) in the ``__LibraryName____``-format given above and are linked in-place (if the input is read from stdin, it is written to stdout). All options except ``--libraries`` are ignored (including ``-o``) in this case. +.. _compiler-api: -***************************************** -Standardized Input and Output Description -***************************************** +Using the Compiler API +********************** The compiler API expects a JSON formatted input and outputs the compilations result in a JSON formatted output. +TBD + +Compiler Input and Output JSON Description +****************************************** + +These JSON formats are used by the compiler API as well as are available through ``solc``. These are subject to change, +some fields are optional (as noted), but it is aimed at to only make backwards compatible changes. + Comments are of course not permitted and used here only for explanatory purposes. Input Description ----------------- -QUESTION: How to specific file-reading callback? - probably not as part of json input - -The input description is language-specific and could change with each compiler version, but it -should be backwards compatible if possible. - .. code-block:: none { @@ -150,8 +156,8 @@ should be backwards compatible if possible. } -Regular Output --------------- +Output Description +------------------ .. code-block:: none From 21a022848540dc488b312f94b62d62fce66aafc7 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 2 Feb 2017 21:10:19 +0000 Subject: [PATCH 15/26] Include pseudo-code of compiler API --- docs/using-the-compiler.rst | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index b89af912d..703a1c577 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -42,7 +42,18 @@ Using the Compiler API The compiler API expects a JSON formatted input and outputs the compilations result in a JSON formatted output. -TBD +See the following pseudo-code: + +.. code-block:: none + + // defined by the consumer of the API + importCallback(url:string) -> content:string + + // invoking the compiler + solc.compile(inputJSON:string, importCallback:function) -> outputJSON:string + +The compiler will ask the ``importCallback`` for each URL defined for a source file and will stop when it succeeds. +If all URLs failed, the compilation results in a failure. Compiler Input and Output JSON Description ****************************************** @@ -66,6 +77,7 @@ Input Description "myFile.sol": { // Optional: keccak256 hash of the source file + // It is used to verify the retrieved content if imported via URLs. "keccak256": "0x123...", // Required (unless "content" is used, see below): URL(s) to the source file. // URL(s) should be imported in this order and the result checked against the From 627a2cec4d058edc6d2e07a37cf0b464ee868b6b Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 2 Feb 2017 21:18:09 +0000 Subject: [PATCH 16/26] Update errors output --- docs/using-the-compiler.rst | 37 ++++++++++++++++++++----------------- 1 file changed, 20 insertions(+), 17 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 703a1c577..2e0cba1f9 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -174,24 +174,27 @@ Output Description .. code-block:: none { - errors: ["error1", "error2"], // we might structure them + // Optional: not present if no errors/warnings were encountered errors: [ - { - // (axic) - file: "sourceFile.sol", // optional? - contract: "contractName", // optional - line: 100, // optional - currently, we always have a byte range in the source file - // Errors/warnings originate in several components, most of them are not - // backend-specific. Currently, why3 errors are part of the why3 output. - // I think it is better to put code-generator-specific errors into the code-generator output - // area, and warnings and errors that are code-generator-agnostic into this general area, - // so that it is easier to determine whether some source code is invalid or only - // triggers errors/warnings in some backend that might only implement some part of solidity. - type: "evm" or "why3" or "ewasm" // maybe a better field name would be needed - severity: "warning" or "error" // mandatory - message: "Invalid keyword" // mandatory - } - ] + { + // Optional + file: "sourceFile.sol", + // Optional + contract: "contractName", + // Optional + line: 100, + // Optional + column: 0, + // Mandatory: Error type, such as "TypeError", "InternalCompilerError", "Exception", etc + type: "TypeError", + // Mandatory: Component where the error originated, such as "general", "why3", "ewasm", etc. + component: "general", + // Mandatory ("error" or "warning") + severity: "error", + // Mandatory + message: "Invalid keyword" + } + ], // This contains all the compiled outputs. It can be limited/filtered by the compilationTarget setting. contracts: { "sourceFile.sol:ContractName": { From 4b19f560b8ac5fd20d46db93a3aa2d1c41adb0db Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 2 Feb 2017 21:27:12 +0000 Subject: [PATCH 17/26] Make contracts output two-level --- docs/using-the-compiler.rst | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 2e0cba1f9..3282b85a0 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -195,13 +195,14 @@ Output Description message: "Invalid keyword" } ], - // This contains all the compiled outputs. It can be limited/filtered by the compilationTarget setting. + // This contains the contract-level outputs. It can be limited/filtered by the outputSelection settings. contracts: { - "sourceFile.sol:ContractName": { - // The Ethereum Contract ABI. If empty, it is represented as an empty array. - // See https://github.com/ethereum/wiki/wiki/Ethereum-Contract-ABI - abi: [], - evm: { + "sourceFile.sol": { + "ContractName": { + // The Ethereum Contract ABI. If empty, it is represented as an empty array. + // See https://github.com/ethereum/wiki/wiki/Ethereum-Contract-ABI + abi: [], + evm: { assembly: bytecode: runtimeBytecode: @@ -219,16 +220,17 @@ Output Description // furthermore, runtime bytecode is always a substring of the bytecode anyway. runtimeLinkReferences: { } - }, - functionHashes: - metadata: // see the Metadata Output documentation - ewasm: { + }, + functionHashes: + metadata: // see the Metadata Output documentation + ewasm: { wast: // S-expression format wasm: // - }, - userdoc: // Obsolete - devdoc: // Obsolete - natspec: // Combined dev+userdoc + }, + userdoc: // Obsolete + devdoc: // Obsolete + natspec: // Combined dev+userdoc + } } }, formal: { From c217bc2dcaf9deec38a54cc094b09288e95ba58f Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 2 Feb 2017 21:33:46 +0000 Subject: [PATCH 18/26] Updated EVM output --- docs/using-the-compiler.rst | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 3282b85a0..2d78d58a0 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -204,22 +204,25 @@ Output Description abi: [], evm: { assembly: - bytecode: - runtimeBytecode: + // Bytecode and related details. + bytecode: { + // The bytecode as a hex string. + object: "00fe", + // The source mapping as a string. See the source mapping definition. + sourceMap: "", + // If given, this is an unlinked object. + linkReferences: { + "libraryFile.sol": { + // Byte offsets into the bytecode. Linking replaces the 20 bytes located there. + "Library1": [1, 200, 80] + } + } + } + // The same layout as above. + deployedBytecode: { }, opcodes: annotatedOpcodes: // (axic) see https://github.com/ethereum/solidity/issues/1178 gasEstimates: - sourceMap: - runtimeSourceMap: - // If given, this is an unlinked object (cannot be filtered out explicitly, might be - // filtered if both bytecode, runtimeBytecode, opcodes and others are filtered out) - linkReferences: { - "sourceFile.sol:Library1": [1, 200, 80] // byte offsets into bytecode. Linking replaces the 20 bytes there. - } - // the same for runtimeBytecode - I'm not sure it is a good idea to allow to link libraries differently for the runtime bytecode. - // furthermore, runtime bytecode is always a substring of the bytecode anyway. - runtimeLinkReferences: { - } }, functionHashes: metadata: // see the Metadata Output documentation From a3340e210e94c3e31cf26b65cc49e03f8ef444de Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 2 Feb 2017 21:35:07 +0000 Subject: [PATCH 19/26] Error list should have sourceLocation --- docs/using-the-compiler.rst | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 2d78d58a0..67b3f32ca 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -179,12 +179,10 @@ Output Description { // Optional file: "sourceFile.sol", - // Optional - contract: "contractName", - // Optional - line: 100, - // Optional - column: 0, + // Optional: Location within the source file. + sourceLocation: [ + { start: 0, end: 100 }, + ], // Mandatory: Error type, such as "TypeError", "InternalCompilerError", "Exception", etc type: "TypeError", // Mandatory: Component where the error originated, such as "general", "why3", "ewasm", etc. From 9fc017d10b3da56f4e94b73bf9f9f6ec7d5345cf Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 2 Feb 2017 21:35:57 +0000 Subject: [PATCH 20/26] Support linkReferences with length specified --- docs/using-the-compiler.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 67b3f32ca..ed7f08565 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -212,7 +212,10 @@ Output Description linkReferences: { "libraryFile.sol": { // Byte offsets into the bytecode. Linking replaces the 20 bytes located there. - "Library1": [1, 200, 80] + "Library1": [ + { start: 0, length: 20 }, + { start: 200, length: 20 } + ] } } } From 9fa54db7bd0003da3ef4f7daa6df860e71f7af75 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 2 Feb 2017 21:52:35 +0000 Subject: [PATCH 21/26] Explain every contract output field --- docs/using-the-compiler.rst | 49 ++++++++++++++++++++++++++++--------- 1 file changed, 38 insertions(+), 11 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index ed7f08565..6a50ca685 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -201,7 +201,14 @@ Output Description // See https://github.com/ethereum/wiki/wiki/Ethereum-Contract-ABI abi: [], evm: { - assembly: + // Intermediate representation (string) + ir: "", + // Assembly (string) + assembly: "", + // Old-style assembly (string) + asm: "", + // Old-style assembly (JSON object) + asmJSON: [], // Bytecode and related details. bytecode: { // The bytecode as a hex string. @@ -221,19 +228,39 @@ Output Description } // The same layout as above. deployedBytecode: { }, - opcodes: - annotatedOpcodes: // (axic) see https://github.com/ethereum/solidity/issues/1178 - gasEstimates: + // Opcodes list (string) + opcodes: "", + // The list of function hashes + methodIdentifiers: { + "5c19a95c": "delegate(address)", + }, + // Function gas estimates + gasEstimates: { + creation: { + dataCost: 420000, + // -1 means infinite (aka. unknown) + executionCost: -1 + }, + external: { + "delegate(address)": 25000 + }, + internal: { + "heavyLifting()": -1 + } + } }, - functionHashes: - metadata: // see the Metadata Output documentation + // See the Metadata Output documentation + metadata: {}, ewasm: { - wast: // S-expression format - wasm: // + // S-expressions format + wast: "", + // Binary format (hex string) + wasm: "" }, - userdoc: // Obsolete - devdoc: // Obsolete - natspec: // Combined dev+userdoc + // User documentation (natspec) + userdoc: {}, + // Developer documentation (natspec) + devdoc: {} } } }, From 96677cd1788ab7eb8c493f4de7030200b0c39108 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 2 Feb 2017 22:06:10 +0000 Subject: [PATCH 22/26] Update the AST output --- docs/using-the-compiler.rst | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 6a50ca685..f2d6b858a 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -193,6 +193,15 @@ Output Description message: "Invalid keyword" } ], + // This contains the file-level outputs. In can be limited/filtered by the outputSelection settings. + sources: { + "sourceFile.sol": { + // Identifier (used in source maps) + id: 1, + // The AST object + ast: {} + } + }, // This contains the contract-level outputs. It can be limited/filtered by the outputSelection settings. contracts: { "sourceFile.sol": { @@ -264,13 +273,6 @@ Output Description } } }, - formal: { - "why3": "..." - }, - sourceList: ["source1.sol", "source2.sol"], // this is important for source references both in the ast as well as in the srcmap in the contract - sources: { - "source1.sol": { - "AST": { ... } - } - } + // Why3 output (string) + why3: "" } From 10d3a591d42952980203c6f9395f81065f06575b Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Mon, 6 Feb 2017 16:55:56 +0000 Subject: [PATCH 23/26] Move file into sourceLocation --- docs/using-the-compiler.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index f2d6b858a..ab8f3d826 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -177,11 +177,11 @@ Output Description // Optional: not present if no errors/warnings were encountered errors: [ { - // Optional - file: "sourceFile.sol", // Optional: Location within the source file. - sourceLocation: [ - { start: 0, end: 100 }, + sourceLocation: { + file: "sourceFile.sol", + start: 0, + end: 100 ], // Mandatory: Error type, such as "TypeError", "InternalCompilerError", "Exception", etc type: "TypeError", From 749db7608b269b4b2dde7649d63f32c87184f2b7 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Wed, 8 Feb 2017 19:00:07 +0000 Subject: [PATCH 24/26] Include language field in the JSON --- docs/using-the-compiler.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index ab8f3d826..9ea1a6abc 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -69,6 +69,8 @@ Input Description .. code-block:: none { + // Required: Source code language, such as "Solidity", "serpent", "lll", "assembly", etc. + language: "Solidity", // Required sources: { @@ -205,6 +207,7 @@ Output Description // This contains the contract-level outputs. It can be limited/filtered by the outputSelection settings. contracts: { "sourceFile.sol": { + // If the language used has no contract names, this field should equal to an empty string. "ContractName": { // The Ethereum Contract ABI. If empty, it is represented as an empty array. // See https://github.com/ethereum/wiki/wiki/Ethereum-Contract-ABI From cd81e58e3be5dd52a138f3ccac4ab2a305979acc Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Wed, 8 Feb 2017 19:16:23 +0000 Subject: [PATCH 25/26] Drop the legacy assembly output --- docs/using-the-compiler.rst | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 9ea1a6abc..f197c663b 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -136,15 +136,14 @@ Input Description // devdoc - Developer documentation (natspec) // userdoc - User documentation (natspec) // metadata - Metadata - // evm.ir - New assembly format before desugaring (not supported atm) - // evm.assembly - New assembly format after desugaring (not supported atm) - // evm.asm - Current assembly format (--asm) - // evm.asmJSON - Current assembly format in JSON (--asm-json) + // evm.ir - New assembly format before desugaring + // evm.assembly - New assembly format after desugaring + // evm.legacyAssemblyJSON - Old-style assembly format in JSON // evm.opcodes - Opcodes list // evm.methodIdentifiers - The list of function hashes // evm.gasEstimates - Function gas estimates - // evm.bytecode - Bytecode (--bin) - // evm.deployedBytecode - Deployed bytecode (--bin-runtime) + // evm.bytecode - Bytecode + // evm.deployedBytecode - Deployed bytecode // evm.sourceMap - Source mapping (useful for debugging) // ewasm.wast - eWASM S-expressions format (not supported atm) // ewasm.wasm - eWASM binary format (not supported atm) @@ -218,9 +217,7 @@ Output Description // Assembly (string) assembly: "", // Old-style assembly (string) - asm: "", - // Old-style assembly (JSON object) - asmJSON: [], + legacyAssemblyJSON: [], // Bytecode and related details. bytecode: { // The bytecode as a hex string. From dc431fe1f6e0ff0ddcf1561b7ef136cc654cf976 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Wed, 8 Feb 2017 19:22:40 +0000 Subject: [PATCH 26/26] Simplify the compiler API section (and remove pseudo code) --- docs/using-the-compiler.rst | 24 ++++++------------------ 1 file changed, 6 insertions(+), 18 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index f197c663b..08f181329 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -37,30 +37,18 @@ If ``solc`` is called with the option ``--link``, all input files are interprete .. _compiler-api: -Using the Compiler API -********************** - -The compiler API expects a JSON formatted input and outputs the compilations result in a JSON formatted output. - -See the following pseudo-code: - -.. code-block:: none - - // defined by the consumer of the API - importCallback(url:string) -> content:string - - // invoking the compiler - solc.compile(inputJSON:string, importCallback:function) -> outputJSON:string - -The compiler will ask the ``importCallback`` for each URL defined for a source file and will stop when it succeeds. -If all URLs failed, the compilation results in a failure. - Compiler Input and Output JSON Description ****************************************** +.. warning:: + + This JSON interface is not yet supported by the Solidity compiler, but will be released in a future version. + These JSON formats are used by the compiler API as well as are available through ``solc``. These are subject to change, some fields are optional (as noted), but it is aimed at to only make backwards compatible changes. +The compiler API expects a JSON formatted input and outputs the compilation result in a JSON formatted output. + Comments are of course not permitted and used here only for explanatory purposes. Input Description