All help welcomed!
I posted a big chunk of code here - but feel free to review only a small section. I will use answers to improve this deployed open source software.
About the code
This code works. But can be improved in code quality. Feedback is highly welcomed!
codespell, ktlint, Android Studio validator were already used and followed where it made sense.
Format
If you make a nontrivial suggestions: please add note that you license your work on GPLv3 license. It is not needed for trivial changes, but for substantial I need this to be able to use your work.
I will mention your SO nick in the commit message and link your answer as thanks. Please specify preferred name and email if you want for commit credit. Or request anonymous contribution.
Explaining the program
Background
StreetComplete is an editor of OpenStreetMap database.
OpenStreetMap is an openly licenced geographic database where objects are represented by
geometries - lines/ways/areas/etc.
key-value pairs called tags describing kind of object. For example
highway=motorwaymarking line as a motorway carriageway centerlinewaterway=rivermarking line as a river centerlinebarrier=gatemarking gate (docs)
Typical StreetComplete edits is adding surface=asphalt or surface=dirt to way representing road - to mark road surface.
Finally getting close
Taginfo project listing is used for publishing a simple .json file listing tags used by a given project.
So, this code presented here will parse source code files of StreetComplete to detect which tags can be added/edited, and will list it in a .json file.
Example, everyone loves examples
For example here is defined quest asking whether public toilet is paid.
Among other parts it has:
override fun applyAnswerTo(answer: Boolean, tags: Tags, timestampEdited: Long) { tags["fee"] = answer.toYesNo() } in this case key fee can be set to either fee=yes or fee=no as toYesNo() cannot return anything else.
There are over 100 other quests, some significantly more complex.
Design decisions
In several places shortcuts were taken to avoid very complicated parsing.
As this is tightly coupled to StreetComplete, many assumptions can be made.
Some places are marked with TODOs and I am not expecting help with them.
Running the code
To run code:
git clone https://github.com/matkoniecz/StreetComplete.git cd StreetComplete git checkout code-review ./gradlew updateTaginfoListing which should work (may require installing gradle/kotlin)
Actual code
If someone is interested in looking at full deployed code - see https://github.com/matkoniecz/Zazolc/blob/taginfo/buildSrc/src/main/java/UpdateTaginfoListingTask.kt (on taginfo branch)
To keep it within character and sanity limits I stripped part of the less relevant code, especially part which hardcoded answers.
It is still fully functional.
import com.charleskorn.kaml.Yaml import com.charleskorn.kaml.YamlConfiguration import kotlinx.ast.common.AstSource import kotlinx.ast.common.ast.Ast import kotlinx.ast.common.ast.AstNode import kotlinx.ast.common.ast.DefaultAstNode import kotlinx.ast.common.ast.DefaultAstTerminal import kotlinx.ast.common.ast.astInfoOrNull import kotlinx.ast.common.klass.KlassDeclaration import kotlinx.ast.common.klass.KlassIdentifier import kotlinx.ast.common.klass.KlassString import kotlinx.ast.common.klass.StringComponentRaw import kotlinx.ast.grammar.kotlin.common.summary import kotlinx.ast.grammar.kotlin.target.antlr.kotlin.KotlinGrammarAntlrKotlinParser import kotlinx.serialization.ExperimentalSerializationApi import kotlinx.serialization.Serializable import kotlinx.serialization.decodeFromString import kotlinx.serialization.encodeToString import kotlinx.serialization.json.Json import org.gradle.api.DefaultTask import org.gradle.api.tasks.Input import org.gradle.api.tasks.TaskAction import java.io.File import java.io.InputStream import kotlin.system.exitProcess /* Generate Taginfo tag listing - only tags added or edited are listed * Tags removed or used in filtering are NOT listed. * * Follows https://wiki.openstreetmap.org/wiki/Taginfo/Projects documentation */ @OptIn(ExperimentalSerializationApi::class) // needed by explicitNulls = false open class UpdateTaginfoListingTask : DefaultTask() { @get:Input var targetDir: String? = null companion object { const val NAME_OF_FUNCTION_EDITING_TAGS = "applyAnswerTo" const val KOTLIN_IMPORT_ROOT_WITH_SLASH_ENDING = "app/src/main/java/" const val QUEST_ROOT_WITH_SLASH_ENDING = "app/src/main/java/de/westnordost/streetcomplete/quests/" const val COUNTRY_METADATA_PATH_WITH_SLASH_ENDING = "app/src/main/assets/country_metadata/" // is it possible to use directly SC constants? // import de.westnordost.streetcomplete.osm.SURVEY_MARK_KEY const val SURVEY_MARK_KEY = "check_date" const val VIBRATING_BUTTON = "traffic_signals:vibration" private const val SOUND_SIGNALS = "traffic_signals:sound" } private fun generateReport(questData: List<TagQuestInfo>) { println(targetDir) val format = Json { encodeDefaults = true; explicitNulls = false; prettyPrint = true } @Serializable data class TagWithDescriptionForTaginfoListing(val key: String, val value: String?, val description: String) @Serializable data class Project(val name: String, val description: String, val project_url: String, val doc_url: String, val icon_url: String, val contact_name: String, val contact_email: String) @Serializable data class TaginfoReport(val data_format: Int = 1, val data_url: String, val project: Project, val tags: List<TagWithDescriptionForTaginfoListing>) fun showChangesMadeIfAny(report: TaginfoReport, oldReport: TaginfoReport) { if (report.tags != oldReport.tags) { println("new tags are different! verify that") report.tags.forEach { if (it !in oldReport.tags) { println("new entry: $it") } } oldReport.tags.forEach { if (it !in report.tags) { println("removed entry: $it") } } } } // https://github.com/Kotlin/kotlinx.serialization/blob/master/docs/json.md val project = Project("StreetComplete", "Surveyor app for Android", "https://github.com/westnordost/StreetComplete", "https://wiki.openstreetmap.org/wiki/StreetComplete", "https://raw.githubusercontent.com/westnordost/StreetComplete/master/app/src/main/res/mipmap-xhdpi/ic_launcher.png", "my name, snipped in SO", "my email, snipped in SO", ) val dataUrl = "snipped in SO due to character limit" val report = TaginfoReport(1, dataUrl, project, questData.map { TagWithDescriptionForTaginfoListing(it.tag.key, it.tag.value, "added or edited tag") } ) val jsonText = format.encodeToString(report) val targetFile = File(targetDir, "taginfo_listing_of_tags_added_or_edited_by_StreetComplete.json") if (targetFile.exists()) { val oldText = targetFile.readText() val oldReport = format.decodeFromString<TaginfoReport>(oldText) showChangesMadeIfAny(report, oldReport) } val fileWriter = targetFile.writer() fileWriter.write(jsonText) fileWriter.close() } @TaskAction fun run() { println(targetDir) val foundTags = mutableListOf<TagQuestInfo>() val folderGenerator = questFolderGenerator() while (folderGenerator.hasNext()) { val folder = folderGenerator.next() File(folder.toString()).walkTopDown().forEach { if (it.isFile) { if (it.name in listOf("AddAddressStreet.kt", "AddRoadName.kt", "AddStreetParking.kt", "AddMaxSpeed.kt", "AddSidewalk.kt", "AddWayLit.kt", "AddMaxWeight.kt", "AddStepsRamp.kt", "AddDrinkingWater.kt", "AddRecyclingContainerMaterials.kt", "AddBuildingType.kt", "AddRoadSurface.kt", "AddPathSurface.kt", "AddFootwayPartSurface.kt", "AddCyclewayPartSurface.kt", "AddStileType.kt", "AddCyclewayWidth.kt", "AddCycleway.kt", "AddSidewalkSurface.kt", "AddPitchSurface.kt", "AddBikeParkingFee.kt", "AddParkingFee.kt", "AddOrchardProduce.kt")) { println("skipping $it") } else if (isQuestFile(it)) { addedOrEditedTags(it)!!.forEach { tags -> foundTags.add(TagQuestInfo(tags, it.name)) } } } } } generateReport(foundTags) reportResultOfDataCollection(foundTags) } private fun questFolderGenerator() = iterator { File(QUEST_ROOT_WITH_SLASH_ENDING).walkTopDown().maxDepth(1).forEach { folder -> if (folder.isDirectory && "$folder/" != QUEST_ROOT_WITH_SLASH_ENDING) { yield(folder) } } } private fun isQuestFile(file: File): Boolean { if (".kt" !in file.name) { return false } listOf("Form.kt", "Adapter.kt", "Utils.kt").forEach { if (it in file.name) { return false } } if (file.name == "AddressStreetAnswer.kt") { return false } if ("Add" in file.name || "Check" in file.name || "Determine" in file.name || "MarkCompleted" in file.name) { return true } return false } private fun reportResultOfDataCollection(foundTags: MutableList<TagQuestInfo>) { println("${foundTags.size} entries registered") val tagsFoundPreviously = 1619 if (foundTags.size != tagsFoundPreviously) { println("Something changed in processing! foundTags count ${foundTags.size} vs $tagsFoundPreviously previously") } } private fun streetCompleteIsReusingAnyValueProvidedByExistingTagging(questDescription: String, key: String): Boolean { // much too complicated and error prone and rare to get that info by parsing if ("MarkCompletedHighwayConstruction" in questDescription && key == "highway") { return true } if ("MarkCompletedBuildingConstruction" in questDescription && key == "building") { return true } return false } private fun freeformKey(key: String): Boolean { // most have own syntax and limitations obeyed by SC // maybe move to general StreetComplete file about OSM tagging? if (key in listOf("name", "int_name", "ref", "addr:flats", "addr:housenumber", "addr:street", "addr:place", "addr:block_number", "addr:streetnumber", "addr:conscriptionnumber", "addr:housename", "building:levels", "roof:levels", "level", "collection_times", "opening_hours", "opening_date", "check_date", "fire_hydrant:diameter", "maxheight", "width", "cycleway:width", "maxspeed", "maxspeed:advisory", "maxstay", "maxweight", "maxweightrating", "maxaxleload", "maxbogieweight", "maxspeed:type", // not really true, but I give up for now. TODO: remove "capacity", "step_count", "lanes", "lanes:forward", "lanes:backward", "lanes:both_ways", "turn:lanes:both_ways", "turn:lanes", "turn:lanes:forward", "turn:lanes:backward", "operator", // technically not fully, but does not make sense to list all that autocomplete values "brand", "sport", // sport=soccer;volleyball is fully valid - doe not entirely fit here but... "produce", // like sport=* )) { return true } if (SURVEY_MARK_KEY in key) { return true } if (key.endsWith(":note")) { return true } if (key.endsWith(":conditional")) { return true } if (key.endsWith(":wikidata")) { return true } if (key.endsWith(":wikipedia")) { return true } if (key.startsWith("lanes:")) { return true } if (key.startsWith("name:")) { return true } if (key.startsWith("source:")) { return true } return false } private fun loadFileText(file: File): String { val inputStream: InputStream = file.inputStream() return inputStream.bufferedReader().use { it.readText() } } @Serializable class Tag(val key: String, val value: String?) { override fun toString(): String { if (value == null) { return "$key=*" } return "$key=$value" } override fun equals(other: Any?): Boolean { if (this === other) return true if (other !is Tag) return false if (key != other.key) return false if (value != other.value) return false return true } override fun hashCode(): Int { var result = key.hashCode() result = 31 * result + (value?.hashCode() ?: 0) return result } } class TagQuestInfo(val tag: Tag, val quest: String) { override fun toString(): String { return "$tag in $quest" } } private fun addedOrEditedTags(file: File): Set<Tag>? { val hardcodedAnswers = addedOrEditedTagsHardcodedAnswers(file) if (hardcodedAnswers != null) { return hardcodedAnswers } val suspectedAnswerEnumFiles = candidatesForEnumFilesForGivenFile(file) val description = file.parentFile.name + File.separator + file.name val fileSourceCode = loadFileText(file) return addedOrEditedTagsRealParsing(description, fileSourceCode, suspectedAnswerEnumFiles) } private fun addedOrEditedTagsHardcodedAnswers(file: File): Set<Tag>? { val fileSourceCode = loadFileText(file) val description = file.parentFile.name + File.separator + file.name var suspectedAnswerEnumFiles = candidatesForEnumFilesForGivenFile(file) // TODO hardcoding is ugly and ideally would be replaced // this function contains cases where answers are partially or fully hardcoded // it is done this way as in some cases parsing would extremely complex and not worth doing this // in some it can be actually implemented and it is likely worth doing this to avoid need // for manual maintenance of the code if ("AddBarrier" in file.name) { // outside when switch to try covering also unlikely new AddBarrier quests // TODO argh? can it be avoided? // why it is present? Without this AddBarrierOnPath would pull also StileTypeAnswer // and claim that barrier=stepover is a thing // would need substantial additional parsing of import data to fix it :( suspectedAnswerEnumFiles = suspectedAnswerEnumFiles.filter { "StileTypeAnswer.kt" !in it.name } return addedOrEditedTagsRealParsing(description, fileSourceCode, suspectedAnswerEnumFiles) } return null } @Serializable data class IncompleteCountryInfo( val additionalStreetsignLanguages: Set<String> = setOf(), val officialLanguages: Set<String> = setOf(), ) private fun possibleLanguageKeys(): MutableSet<String> { val languageTags = mutableSetOf("name", "int_name") File(COUNTRY_METADATA_PATH_WITH_SLASH_ENDING).walkTopDown().maxDepth(1).forEach { file -> if (file.isFile) { val test = Yaml(configuration = YamlConfiguration(strictMode = false)).decodeFromString(IncompleteCountryInfo.serializer(), loadFileText(file)) val langs = test.officialLanguages + test.additionalStreetsignLanguages // if country has single language for street names (langs.size = 1) then // it is using only name tag // if multiple languages are present then this languages are tagged with // name:$langCode tags, such as name:en for English language names if (langs.size > 1) { // international counts for purposes of triggering multi-language support // but itself is rather tagged with int_name tag and listed above already langs.filter { it != "international" }.forEach { langCode -> languageTags.add("name:$langCode") } } } } return languageTags } private fun candidatesForEnumFilesForGivenFile(file: File): List<File> { val suspectedAnswerEnumFilesBasedOnFolder = candidatesForEnumFilesBasedOnFolder(file.parentFile) return suspectedAnswerEnumFilesBasedOnFolder + candidatesForEnumFilesBasedOnImports(file) } private fun candidatesForEnumFilesBasedOnFolder(folder: File): List<File> { val suspectedAnswerEnumFiles = mutableListOf<File>() File(folder.toString()).walkTopDown().forEach { if (isLikelyAnswerEnumFile(it)) { suspectedAnswerEnumFiles.add(it) } } return suspectedAnswerEnumFiles } private fun candidatesForEnumFilesBasedOnImports(file: File): List<File> { // initially just files from folder were taken as a base // due to cases like AddCrossing reaching across folders // it was not working well and require this extra parsing // // also, just parsing imports is not sufficient // see AddBikeParkingType which is not explicitly // importing the enum // // note: importedByFile may have false negatives that require extra parsing // to handle this return importedByFile(file) .filter { isLikelyAnswerEnumFile(File(it)) } .map { File(it) } .filter { it.isFile } } private fun importedByFile(file: File): Set<String> { val returned = mutableSetOf<String>() file.parse().locateByDescription("importList").forEach { importList -> importList.locateByDescription("importHeader").forEach { if (it is DefaultAstNode) { areDirectChildrenMatchingStructureThrowExceptionIfNot("checking import file structure for $path", listOf(listOf("IMPORT", "WS", "identifier", "semi")), it, eraseWhitespace = false) val imported = it.locateSingleOrExceptionByDescriptionDirectChild("identifier") val identifier = imported.locateByDescriptionDirectChild("simpleIdentifier") val pathsFromImportRoot = identifier.joinToString("/") { partBetweenDots -> (partBetweenDots.tree() as KlassIdentifier).identifier } + ".kt" val importedPath = KOTLIN_IMPORT_ROOT_WITH_SLASH_ENDING + pathsFromImportRoot if (File(importedPath).isFile) { // TODO WARNING HACK: false positives here can be expected // TODO WARNING HACK: this will treat // import de.westnordost.streetcomplete.data.user.achievements.EditTypeAchievement.PEDESTRIAN // as import of PEDESTRIAN.kt file // not as import of PEDESTRIAN from EditTypeAchievement.kt file // and this check will result in false negatives in turn... returned.add(importedPath) } } } } return returned } private fun isLikelyAnswerEnumFile(file: File): Boolean { // answers true if it is likely to contain an enum class like // java/de/westnordost/streetcomplete/quests/barrier_type/BarrierType.kt // contains // // TODO: maybe read file source code and simply check is "enum class" text there? if (".kt" !in file.name) { return false } val banned = listOf("SelectPuzzle.kt", "Form.kt", "Util.kt", "Utils.kt", "Adapter.kt", "Drawable.kt", "Dao.kt", "Dialog.kt", "Item.kt", "RotateContainer.kt") banned.forEach { if (it in file.name) { return false } } listOf("OsmFilterQuestType.kt", "MapDataWithGeometry.kt", "Element.kt", "Tags.kt", "OsmElementQuestType.kt", "CountryInfos.kt").forEach { if (it == file.name) { return false } } return !isQuestFile(file) } private fun addedOrEditedTagsRealParsing(description: String, fileSourceCode: String, suspectedAnswerEnumFiles: List<File>): Set<Tag>? { val ast = AstSource.String(description, fileSourceCode).parse() val defaultFunction = ast.extractFunctionByName(NAME_OF_FUNCTION_EDITING_TAGS)!! val functionSourceCode = defaultFunction.relatedSourceCode(fileSourceCode) if ("answer.applyTo(" !in functionSourceCode && "answer.litStatus.applyTo" !in functionSourceCode) { return addedOrEditedTagsWithGivenFunction(description, fileSourceCode, NAME_OF_FUNCTION_EDITING_TAGS, suspectedAnswerEnumFiles) } else { suspectedAnswerEnumFiles.forEach { fileHopefullyWithApplyTo -> val found = fileHopefullyWithApplyTo.parse().extractFunctionByName("applyTo") if (found != null) { // OK, so we found related file providing applyTo function. Great! if ("ParkingFee" in description) { println("$description fpund apply to file $fileHopefullyWithApplyTo") } val got = addedOrEditedTagsRealParsingFindRealEditFunctionViaApplyToFunction(description, fileHopefullyWithApplyTo, suspectedAnswerEnumFiles) val bonusScan = addedOrEditedTagsWithGivenFunction(description, fileSourceCode, NAME_OF_FUNCTION_EDITING_TAGS, suspectedAnswerEnumFiles) if (bonusScan != null && bonusScan.isNotEmpty()) { println(bonusScan) throw ParsingInterpretationException("turns out to be needed") } if (got != null) { return got } } } } return null } private fun addedOrEditedTagsRealParsingFindRealEditFunctionViaApplyToFunction(description: String, fileWithRedirectedFunction: File, suspectedAnswerEnumFiles: List<File>): Set<Tag>? { val found = fileWithRedirectedFunction.parse().extractFunctionByName("applyTo")!! val parameters = found.locateSingleOrExceptionByDescriptionDirectChild("functionValueParameters") .locateByDescriptionDirectChild("functionValueParameter") if (parameters.isEmpty()) { throw ParsingInterpretationException("unsupported") } val parametersInCalledFunction = mutableListOf<String>() for (element in parameters) { val parameter = element.locateSingleOrExceptionByDescriptionDirectChild("parameter") val parameterTree = parameter.tree() if (parameterTree is KlassIdentifier) { parametersInCalledFunction.add(parameterTree.identifier) } else { throw ParsingInterpretationException("should not happen") } } if (parameters.size > 1) { println("Attempting to decompose function call where tags will be modified in another place and multiple arguments were passed") println("$description - parametersInCalledFunction in file ${fileWithRedirectedFunction.name} $parametersInCalledFunction") throw ParsingInterpretationException("No support yet") } return if (parametersInCalledFunction[0] == "tags") { val replacementFunctionName = "applyTo" val replacementSourceCode = loadFileText(fileWithRedirectedFunction) val replacementDescription = fileWithRedirectedFunction.toString() addedOrEditedTagsWithGivenFunction(replacementDescription, replacementSourceCode, replacementFunctionName, suspectedAnswerEnumFiles) } else { // variable is not really supported within called function throw ParsingInterpretationException("redirected function, not using tags variable - unsupported TODO, exiting") } } private fun addedOrEditedTagsWithGivenFunction(description: String, fileSourceCode: String, relevantFunctionName: String, suspectedAnswerEnumFiles: List<File>): Set<Tag>? { val ast = AstSource.String(description, fileSourceCode).parse() val relevantFunction = ast.extractFunctionByName(relevantFunctionName) if (relevantFunction == null) { println(description) println(fileSourceCode) throw ParsingInterpretationException("$relevantFunctionName missing in code provided via $description!") } val appliedTags = mutableSetOf<Tag>() var failedExtraction = false val got = extractCasesWhereTagsAreAccessedWithIndex(description, relevantFunction, fileSourceCode, suspectedAnswerEnumFiles) appliedTags += got appliedTags += extractCasesWhereTagsAreAccessedWithFunction(description, relevantFunction, fileSourceCode, suspectedAnswerEnumFiles) val tagsThatShouldBeMoreSpecific = appliedTags .filter { it.value == null && !freeformKey(it.key) && !streetCompleteIsReusingAnyValueProvidedByExistingTagging(description, it.key) } if (tagsThatShouldBeMoreSpecific.isNotEmpty()) { tagsThatShouldBeMoreSpecific.forEach { println(it) } println("$description found tags which are not freeform but have no specified values") failedExtraction = true } if (appliedTags.size == 0) { return null // parsing definitely failed } if (failedExtraction) { return null } return appliedTags } private fun extractTextFromHardcodedString(passedTextHolder: Ast): String? { var textHolder = passedTextHolder val plausibleText = textHolder.locateByDescription("stringLiteral") if (plausibleText.size == 1) { val textFoundIfFillingEntireHolder = plausibleText[0] if (textHolder.codeRange() == textFoundIfFillingEntireHolder.codeRange()) { // actual text holder is hidden inside, but it is actually the same object val expectedTextHolder = textFoundIfFillingEntireHolder.tree() if (expectedTextHolder is KlassString) { textHolder = expectedTextHolder } } } if (textHolder is KlassString) { if (textHolder.children.size == 1) { val expectedText = textHolder.children[0] if (expectedText is StringComponentRaw) { return expectedText.string } } } return null } private fun extractCasesWhereTagsAreAccessedWithIndex(description: String, relevantFunction: AstNode, fileSourceCode: String, suspectedAnswerEnumFiles: List<File>): Set<Tag> { // it is trying to detect things like // tags["bollard"] = answer.osmValue val appliedTags = mutableSetOf<Tag>() relevantFunction.locateByDescription("assignment").forEach { assignment -> assignment.children.forEach { tagsDictAccess -> if (assignsToTagsVariable(tagsDictAccess)) { // this limits it to things like // tags[something] = somethingElse // (would it also detect tags=whatever)? val indexingElement = tagsDictAccess.locateSingleOrExceptionByDescription("assignableSuffix") .locateSingleOrExceptionByDescription("indexingSuffix") // indexingElement is something like ["indoor"] or [key] val expression = indexingElement.locateSingleOrExceptionByDescriptionDirectChild("expression") // drop outer [ ] val potentialTexts = expression.locateByDescription("stringLiteral", debug = false) // what if it is something like "prefix" + CONSTANT ? val potentiallyUsableExpression = if (expression is KlassIdentifier) { expression } else { null } // val likelyVariable = expression.locateByDescriptionDirectChild("disjunction") // tag[key] = ... for example if (potentialTexts.size == 1) { val processed = potentialTexts[0].tree() if (processed == null) { throw ParsingInterpretationException("not handled") } val key = extractTextFromHardcodedString(processed) if (key == null) { throw ParsingInterpretationException("key not found") } else { val valueHolder = assignment.locateSingleOrExceptionByDescriptionDirectChild("expression") appliedTags += extractValuesForKnownKey(description, key, valueHolder, fileSourceCode, suspectedAnswerEnumFiles) } } else if (potentiallyUsableExpression != null) { throw ParsingInterpretationException("expression in identified access as a variable") } else if (likelyVariable.size == 1) { if (likelyVariable[0].relatedSourceCode(fileSourceCode) == "key" && "name:\$languageTag" in fileSourceCode) { // special handling for name quests possibleLanguageKeys().forEach { appliedTags.add(Tag(it, null)) } } else { throw ParsingInterpretationException("expression in identified access as a complex variable") } } else { throw ParsingInterpretationException("not handled, ${potentialTexts.size} texts, $potentiallyUsableExpression variable") } } } } return appliedTags } private fun assignsToTagsVariable(tagsDictAccess: Ast): Boolean { return tagsDictAccess.description == "directlyAssignableExpression" && tagsDictAccess is DefaultAstNode && tagsDictAccess.children[0].tree() is KlassIdentifier && ((tagsDictAccess.children[0].tree() as KlassIdentifier).identifier == "tags") } class EnumFieldState(val identifier: String, val possibleValue: String) { // entry such as // osmKey = building // from // HOUSE ("building", "house"), // from // enum class BuildingType(val osmKey: String, val osmValue: String) { override fun toString(): String { return "EnumFieldState($identifier, $possibleValue)" } } class EnumEntry(val identifier: String, val fields: List<EnumFieldState>) { // entry such as // HOUSE ("building", "house"), // from // enum class BuildingType(val osmKey: String, val osmValue: String) { override fun toString(): String { return "EnumEntry($identifier, $fields)" } } private fun getEnumValuesDefinedInThisFile(description: String, file: File, debug: Boolean = false): Set<EnumEntry> { val values = mutableSetOf<EnumEntry>() val fileMaybeContainingEnumSourceCode = loadFileText(file) val potentialEnumFileAst = file.parse() var enumsTried = 0 potentialEnumFileAst.locateByDescription("classDeclaration").forEach { enum -> val modifiers = enum.locateByDescription("modifiers") if (modifiers.size != 1) { // not expected to be enum // will happen if potential enum file contains rather class such as // class StreetSideSelectRotateContainer @JvmOverloads constructor( return@forEach // skip silently as heuristic being too eager } else if (modifiers[0].relatedSourceCode(fileMaybeContainingEnumSourceCode) == "enum") { enumsTried += 1 val enumFieldNames = mutableListOf<String>() val constructor = enum.locateSingleOrNullByDescription("primaryConstructor") if (constructor == null) { // may happen with helper enums being present, such as // enum class FireHydrantDiameterMeasurementUnit { MILLIMETER, INCH } return@forEach // skip silently as heuristic being too eager } constructor.locateSingleOrExceptionByDescriptionDirectChild("classParameters") .locateByDescriptionDirectChild("classParameter") .forEach { val simpleIdentifier = it.locateSingleOrExceptionByDescriptionDirectChild("simpleIdentifier") .relatedSourceCode(fileMaybeContainingEnumSourceCode) enumFieldNames.add(simpleIdentifier) } enum.locateByDescription("enumEntry").forEach { enumEntry -> var extractedText: String? val identifier = (enumEntry.locateSingleOrNullByDescriptionDirectChild("simpleIdentifier")!!.tree() as KlassIdentifier).identifier val valueArguments = enumEntry.locateSingleOrNullByDescriptionDirectChild("valueArguments") if (valueArguments == null) { val explanation = "parsing ${file.path} failed, valueArguments count is not 1, skipping, maybe it should be also investigated" throw ParsingInterpretationException(explanation) } else { val enumFieldGroup = mutableListOf<EnumFieldState>() val arguments = valueArguments.locateByDescriptionDirectChild("valueArgument") for (i in arguments.indices) { extractedText = extractTextFromHardcodedString(arguments[i]) if (extractedText == null) { if (arguments[i].tree() is KlassDeclaration && (arguments[i].tree() as KlassDeclaration).identifier.toString() == "null") { // it has null as value, apparently // lest skip it silently } else { throw ParsingInterpretationException("showHumanReadableTreeWithSourceCode(fileMaybeContainingEnumSourceCode) - showing ${file.path} after enum extraction failed") } } else { enumFieldGroup.add(EnumFieldState(enumFieldNames[i], extractedText)) } } if (enumFieldGroup.size > 0) { values.add(EnumEntry(identifier, enumFieldGroup)) } } } } } if (values.size == 0 && debug) { println("enum extraction from ${file.path} failed! $enumsTried potential enums tried ($description request)") } return values } private fun extractValuesForKnownKey(description: String, key: String, valueHolder: Ast, fileSourceCode: String, suspectedAnswerEnumFiles: List<File>): MutableSet<Tag> { val appliedTags = mutableSetOf<Tag>() var scanned: MutableSet<Tag>? scanned = extractValuesForKnownKeyFromWhenExpressionIfSingleOneIsPresent(description, key, valueHolder, fileSourceCode, suspectedAnswerEnumFiles) if (scanned != null) { return scanned } scanned = extractValuesForKnownKeyFromIfExpressionIfSingleOneIsPresent(description, key, valueHolder, fileSourceCode, suspectedAnswerEnumFiles) if (scanned != null) { return scanned } val valueIfItIsSimpleText = extractTextFromHardcodedString(valueHolder) val valueHolderSourceCode = valueHolder.relatedSourceCode(fileSourceCode) if (valueIfItIsSimpleText != null) { appliedTags.add(Tag(key, valueIfItIsSimpleText)) } else if (valueHolderSourceCode.endsWith(".toYesNo()")) { appliedTags.add(Tag(key, "yes")) appliedTags.add(Tag(key, "no")) } else if (valueHolderSourceCode.endsWith(".toCheckDateString()")) { appliedTags.add(Tag(key, null)) } else if (valueHolderSourceCode == "answer.joinToString(\";\") { it.osmValue }") { // answer.joinToString(";") { it.osmValue } val filtered = valueHolder.locateSingleOrExceptionByDescription("lambdaLiteral").locateSingleOrExceptionByDescriptionDirectChild("statements") appliedTags += provideTagsBasedOnAswerDataStructuresFromExternalFiles(description, key, filtered, suspectedAnswerEnumFiles) appliedTags.add(Tag(key, null)) // as it can be joined in basically any combination and listing all permutations would be absurd. Maybe provide comment here of taginfo listing supports this? } else if (valueHolderSourceCode.startsWith("answer.") || valueHolderSourceCode.startsWith("this.")) { appliedTags += provideTagsBasedOnAswerDataStructuresFromExternalFiles(description, key, valueHolder, suspectedAnswerEnumFiles) } else { if ( freeformKey(key) || streetCompleteIsReusingAnyValueProvidedByExistingTagging(description, key)) { appliedTags.add(Tag(key, null)) } else { val explanation = "exact value is missing, extractValuesForKnownKey failed. $description get value (key is known: $key) from <$valueHolderSourceCode> somehow..." throw ParsingInterpretationException(explanation) } } return appliedTags } private fun provideTagsBasedOnAswerDataStructuresFromExternalFiles(description: String, key: String, valueHolder: Ast, suspectedAnswerEnumFiles: List<File>, debug: Boolean = false): MutableSet<Tag> { val appliedTags = mutableSetOf<Tag>() var extractedSomething = false suspectedAnswerEnumFiles.forEach { getEnumValuesDefinedInThisFile(description, it).forEach { enumGroup -> enumGroup.fields.forEach { value -> // why redefined in each cycle? // because there are cases where it would fail - but these are also cases // where extracting enum also fails, so is not triggered and can be ignored val postfixUnarySuffixes = valueHolder.locateByDescription("postfixUnarySuffix") if (postfixUnarySuffixes.size != 1) { throw ParsingInterpretationException("$key values extraction in provideTagsBasedOnAswerDataStructuresFromExternalFiles - postfixUnarySuffix expected to be a single one, got ${postfixUnarySuffixes.size}") } val accessIdentifierAst = postfixUnarySuffixes[0] .locateSingleOrExceptionByDescriptionDirectChild("navigationSuffix") .locateSingleOrExceptionByDescriptionDirectChild("simpleIdentifier") val identifier = (accessIdentifierAst.tree() as KlassIdentifier).identifier if (value.identifier == identifier) { appliedTags.add(Tag(key, value.possibleValue)) extractedSomething = true if (debug) { println("$key=${value.possibleValue} registered based on ${value.identifier} identifier matching expected $identifier - from ${it.name}") } } } } } if (!freeformKey(key)) { // with freeform keys heuristic below will just get // variable such as capacity and will get confused // It is possible to get it working but not worth it right now suspectedAnswerEnumFiles.forEach { file -> val code = loadFileText(file) val ast = file.parse() val classDeclarations = ast.locateByDescription("classDeclaration") if (classDeclarations.isEmpty()) { return@forEach } ast.locateByDescription("propertyDeclaration").forEach { astNode -> val whenExpression = astNode.locateSingleOrNullByDescription("whenExpression") if (whenExpression != null) { extractValuesForKnownKeyFromWhenExpression(description, "dummykey", whenExpression, code, listOf()).forEach { if (debug) { println("OBTAINED FROM WHEN IN CLASS DECLARATION! $description $key=${it.value}") } appliedTags.add(Tag(key, it.value)) extractedSomething = true } } } } } if (!extractedSomething) { appliedTags.add(Tag(key, null)) if ( freeformKey(key) || streetCompleteIsReusingAnyValueProvidedByExistingTagging(description, key)) { // no reason to complain } else { throw ParsingInterpretationException("failed to find values for now - key is $key") } } return appliedTags } private fun extractValuesForKnownKeyFromIfExpressionIfSingleOneIsPresent(description: String, key: String, valueHolder: Ast, fileSourceCode: String, suspectedAnswerEnumFiles: List<File>): MutableSet<Tag>? { val ifExpression = valueHolder.locateSingleOrNullByDescription("ifExpression") if (ifExpression != null) { if (ifExpression.relatedSourceCode(fileSourceCode) == valueHolder.relatedSourceCode(fileSourceCode)) { return extractValuesForKnownKeyFromIfExpression(description, key, ifExpression, fileSourceCode, suspectedAnswerEnumFiles) } else { throw ParsingInterpretationException("not handled, when expressions as part of something bigger") } } return null } private fun extractValuesForKnownKeyFromIfExpression(description: String, key: String, ifExpression: AstNode, fileSourceCode: String, suspectedAnswerEnumFiles: List<File>): MutableSet<Tag> { val appliedTags = mutableSetOf<Tag>() ifExpression.locateByDescription("controlStructureBody").forEach { appliedTags += extractValuesForKnownKey(description, key, it, fileSourceCode, suspectedAnswerEnumFiles) } return appliedTags } private fun extractValuesForKnownKeyFromWhenExpressionIfSingleOneIsPresent(description: String, key: String, valueHolder: Ast, fileSourceCode: String, suspectedAnswerEnumFiles: List<File>): MutableSet<Tag>? { val whenExpression = valueHolder.locateSingleOrNullByDescription("whenExpression") if (whenExpression != null) { if (whenExpression.relatedSourceCode(fileSourceCode) == valueHolder.relatedSourceCode(fileSourceCode)) { return extractValuesForKnownKeyFromWhenExpression(description, key, whenExpression, fileSourceCode, suspectedAnswerEnumFiles) } else { throw ParsingInterpretationException("not handled, when expressions as part of something bigger") } } return null } private fun extractValuesForKnownKeyFromWhenExpression(description: String, key: String, whenExpression: AstNode, fileSourceCode: String, suspectedAnswerEnumFiles: List<File>): MutableSet<Tag> { val appliedTags = mutableSetOf<Tag>() whenExpression.locateByDescription("whenEntry").forEach { it -> val structure = it.children.filter { it.description != "WS" } val expectedStructureA = listOf("whenCondition", "ARROW", "controlStructureBody", "semi") val expectedStructureB = listOf("ELSE", "ARROW", "controlStructureBody", "semi") areDirectChildrenMatchingStructureThrowExceptionIfNot(description, listOf(expectedStructureA, expectedStructureB), it, eraseWhitespace = true) appliedTags += extractValuesForKnownKey(description, key, structure[2], fileSourceCode, suspectedAnswerEnumFiles) } return appliedTags } private fun areDirectChildrenMatchingStructureThrowExceptionIfNot(description: String, expectedStructures: List<List<String>>, expression: AstNode, eraseWhitespace: Boolean) { val structure = expression.children.filter { !(eraseWhitespace && it.description == "WS") }.map { it.description } expectedStructures.forEach { if (it == structure) { return } } var maxLength = 0 expectedStructures.forEach { if (maxLength < it.size) { maxLength = it.size } } for (i in 0 until maxLength) { expectedStructures.forEach { if (it.size > i) { if (it[i] != structure[i]) { throw ParsingInterpretationException("unexpected structure! at $i index") } } } } } private fun surveyMarkKeyBasedOnKey(key: String): String { // TODO - can we directly call relevant StreetComplete code? return "$SURVEY_MARK_KEY:$key" } private fun extractCasesWhereTagsAreAccessedWithFunction(description: String, relevantFunction: AstNode, fileSourceCode: String, suspectedAnswerEnumFiles: List<File>): Set<Tag> { // it is trying to detect things like // tags.updateWithCheckDate("smoking", answer.osmValue) val appliedTags = mutableSetOf<Tag>() relevantFunction.locateByDescription("postfixUnaryExpression") .filter { isAccessingTagsVariableWithMemberFunction(it) } .forEach { accessingTagsWithFunction -> val dotAndFunction = accessingTagsWithFunction.locateByDescriptionDirectChild("postfixUnarySuffix")[0].locateSingleOrExceptionByDescriptionDirectChild("navigationSuffix") if (dotAndFunction !is AstNode) { throw ParsingInterpretationException("unexpected!") } val functionName = getNameOfFunctionFromNavigationSuffix(dotAndFunction) if (functionName in listOf( "setCheckDateForKey", "updateCheckDateForKey" ) ) { // only check data for val keyString = extractStringLiteralArgumentInFunctionCall(description, 0, accessingTagsWithFunction) if (keyString != null) { appliedTags.add(Tag(surveyMarkKeyBasedOnKey(keyString), null)) } } else if (functionName == "updateWithCheckDate") { var keyString = extractStringLiteralArgumentInFunctionCall(description, 0, accessingTagsWithFunction) val valueString = extractStringLiteralArgumentInFunctionCall(description, 1, accessingTagsWithFunction) // fold it into extractArgumentInFunctionCall? // try to automatically obtain this constants? if (keyString == null) { val keyArgumentAst = extractArgumentSyntaxTreeInFunctionCall(0, accessingTagsWithFunction).locateSingleOrNullByDescription("primaryExpression") if (keyArgumentAst == null) { throw ParsingInterpretationException("unexpected") } val keyArgumentAstTree = keyArgumentAst.tree() if (keyArgumentAstTree is KlassIdentifier) { if (keyArgumentAstTree.identifier == "SOUND_SIGNALS") { keyString = SOUND_SIGNALS } if (keyArgumentAstTree.identifier == "VIBRATING_BUTTON") { keyString = VIBRATING_BUTTON } } } if (keyString != null) { appliedTags.add(Tag("$SURVEY_MARK_KEY:$keyString", null)) if (valueString != null) { appliedTags.add(Tag(keyString, valueString)) } else { val valueAst = extractArgumentSyntaxTreeInFunctionCall(1, accessingTagsWithFunction) val valueHolderSourceCode = valueAst.relatedSourceCode(fileSourceCode) if (valueHolderSourceCode == "answer.toYesNo()") { // kind of hackish, fix this? appliedTags.add(Tag(keyString, "yes")) appliedTags.add(Tag(keyString, "no")) } else if (valueHolderSourceCode == "answer.osmValue" || valueHolderSourceCode == "answer.value.osmValue") { val dotAcess = valueAst.locateByDescription("postfixUnarySuffix") if (dotAcess.isEmpty()) { throw ParsingInterpretationException("hmmmmmmmm") } val accessIdentifierAst = dotAcess[dotAcess.size - 1].locateSingleOrExceptionByDescriptionDirectChild("navigationSuffix") .locateSingleOrExceptionByDescriptionDirectChild("simpleIdentifier") val identifier = (accessIdentifierAst.tree() as KlassIdentifier).identifier var extractedNothing = true suspectedAnswerEnumFiles.forEach { getEnumValuesDefinedInThisFile(description, it).forEach { value -> // dotAcess will have a single element [.osmValue] on "answer.osmValue" // dotAcess will have a two elements [.value, .osmValue] on "answer.value.osmValue" if (value.fields.size != 1) { throw ParsingInterpretationException("expected a single value, got $value") } if (value.fields[0].identifier == identifier) { appliedTags.add(Tag(keyString, value.fields[0].possibleValue)) } extractedNothing = false } } if (extractedNothing) { throw ParsingInterpretationException("Enum obtaining failed! suspectedAnswerEnumFiles $suspectedAnswerEnumFiles") } } else { val valueSourceCode = valueAst.relatedSourceCode(fileSourceCode) if (freeformKey(keyString) && valueSourceCode in setOf("answer.toString()", "openingHoursString", "answer.times.toString()", "duration.toOsmValue()", "toOsmValue()")) { // key is freeform and it appears to not be enum - so lets skip complaining and attempting to tarck down value // individual quests can be investigated as needed appliedTags.add(Tag(keyString, null)) } else { throw ParsingInterpretationException("extractCasesWhereTagsAreAccessedWithFunction - extraction failing: $description tags dict is accessed with updateWithCheckDate, key known ($keyString), value unknown, obtaining data failed") } } } } else { throw ParsingInterpretationException("$description - failed to extract key from updateWithCheckDate") } } else if (functionName in listOf("remove", "containsKey", "removeCheckDatesForKey", "hasChanges", "entries", "hasCheckDateForKey", "hasCheckDate")) { // skip, as only added or edited tags are listed - and removed one and influencing ones are ignored } else if (functionName in listOf("updateCheckDate")) { appliedTags.add(Tag(SURVEY_MARK_KEY, null)) } else if (functionName == "replaceShop") { // parsing skipped per // https://github.com/streetcomplete/StreetComplete/issues/4225#issuecomment-1190487094 } else { throw ParsingInterpretationException("unexpected function name $functionName in $description") } } return appliedTags } private fun isAccessingTagsVariableWithMemberFunction(ast: AstNode): Boolean { val root = ast.tree() if (root !is KlassIdentifier) { return false } if (root.identifier != "tags") { return false } val primary = ast.locateSingleOrExceptionByDescriptionDirectChild("primaryExpression") val rootOfExpectedTagsIdentifier = primary.tree() if (rootOfExpectedTagsIdentifier !is KlassIdentifier) { throw ParsingInterpretationException("unexpected! primary is ${primary::class}") } if (rootOfExpectedTagsIdentifier.identifier != "tags") { throw ParsingInterpretationException("unexpected!") } val possibleDotAndFunction = ast.locateByDescriptionDirectChild("postfixUnarySuffix") if (possibleDotAndFunction.isEmpty()) { // this will happen in case of say // tags["key"] = value // in such case we want to skip it return false } val expectedToHoldDotAndFunctionCall = possibleDotAndFunction[0].locateByDescriptionDirectChild("navigationSuffix") if (expectedToHoldDotAndFunctionCall.isEmpty()) { // maybe false positive? // maybe something like // .any { tags[it]?.toCheckDate() != null } // where skipping is valid? return false } return true } private fun extractArgumentListSyntaxTreeInFunctionCall(ast: AstNode): List<AstNode> { val arguments = ast.locateByDescriptionDirectChild("postfixUnarySuffix")[1] .locateSingleOrExceptionByDescriptionDirectChild("callSuffix") .locateSingleOrExceptionByDescriptionDirectChild("valueArguments") return arguments.locateByDescription("valueArgument") } private fun extractArgumentSyntaxTreeInFunctionCall(index: Int, ast: AstNode): AstNode { return extractArgumentListSyntaxTreeInFunctionCall(ast)[index] } private fun extractStringLiteralArgumentInFunctionCall(description: String, index: Int, ast: AstNode): String? { val found = extractArgumentSyntaxTreeInFunctionCall(index, ast).locateSingleOrNullByDescription("primaryExpression") if (found == null) { throw ParsingInterpretationException("war") } if (found.children.size == 1) { return if (found.children[0].description == "stringLiteral") { val stringObject = (found.children[0].tree() as KlassString).children[0] (stringObject as StringComponentRaw).string } else { // as function name mentions, only string literals will be recovered null } } else { throw ParsingInterpretationException("unhandled extraction of $index function parameter - multiple children") } } private fun getNameOfFunctionFromNavigationSuffix(dotAndFunction: AstNode): String { if (dotAndFunction.description != "navigationSuffix") { exitProcess(1) } val expectedPackagedDot = dotAndFunction.children[0] if (expectedPackagedDot.description != "memberAccessOperator") { throw ParsingInterpretationException("unexpected!") } if (expectedPackagedDot !is AstNode) { throw ParsingInterpretationException("unexpected!") } val expectedDot = expectedPackagedDot.children[0] if (expectedDot !is DefaultAstTerminal) { throw ParsingInterpretationException("unexpected!") } if (expectedDot.text != ".") { throw ParsingInterpretationException("unexpected!") } val expectedFunctionIdentifier = dotAndFunction.children[1] if (expectedFunctionIdentifier.description != "simpleIdentifier") { throw ParsingInterpretationException("unexpected!") } if (expectedFunctionIdentifier.tree() !is KlassIdentifier) { throw ParsingInterpretationException("unexpected! expectedFunctionIdentifier.root() is ${expectedFunctionIdentifier.tree()!!::class}") } return (expectedFunctionIdentifier.tree() as KlassIdentifier).identifier } class ParsingInterpretationException(private val s: String) : Throwable() { override fun toString(): String { return s } } private fun Ast.codeRange(): Pair<Int, Int> { val start = tree()!!.astInfoOrNull!!.start.index val end = tree()!!.astInfoOrNull!!.stop.index return Pair(start, end) } private fun Ast.relatedSourceCode(sourceCode: String): String { if (tree() == null) { return "<source code not available>" } val start = tree()!!.astInfoOrNull!!.start.index val end = tree()!!.astInfoOrNull!!.stop.index if (start < 0 || end < 0) { return "<source code not available> - stated range was $start to $end index" } return sourceCode.subSequence(start, end).toString() } private fun Ast.locateSingleOrNullByDescription(filter: String, debug: Boolean = false): AstNode? { val found = locateByDescription(filter, debug) return if (found.size != 1) { null } else { found[0] } } private fun Ast.locateSingleOrExceptionByDescription(filter: String, debug: Boolean = false): AstNode { val found = locateByDescription(filter, debug) if (found.size != 1) { throw ParsingInterpretationException("unexpected count! Expected single matching on filter $filter, got ${found.size}") } else { return found[0] } } private fun Ast.locateByDescription(filter: String, debug: Boolean = false): List<AstNode> { if (this is AstNode) { val fromChildren = children.flatMap { child -> child.locateByDescription(filter, debug) } return if (description == filter) { if (debug) { println("$filter filter matching description") } listOf(this) + fromChildren } else { if (debug) { println("$filter filter NOT matching description $description") } fromChildren } } else { return listOf() } } private fun Ast.locateSingleOrExceptionByDescriptionDirectChild(filter: String): Ast { val found = locateByDescriptionDirectChild(filter) if (found.size != 1) { throw ParsingInterpretationException("unexpected count! Expected single matching direct child on filter $filter, got ${found.size}") } else { return found[0] } } private fun Ast.locateSingleOrNullByDescriptionDirectChild(filter: String): Ast? { val found = locateByDescriptionDirectChild(filter) return if (found.size != 1) { null } else { found[0] } } private fun Ast.locateByDescriptionDirectChild(filter: String): List<Ast> { val returned = mutableListOf<Ast>() if (this is AstNode) { for (child in children) { if (child.description == filter) { returned.add(child) } } } return returned } private fun Ast.extractFunctionByName(functionName: String): AstNode? { val got = extractAllFunctionsByName(functionName) if (got.size > 1) { throw ParsingInterpretationException("expected one function, got multiple") } if (got.isEmpty()) { return null } return got[0] } private fun Ast.extractAllFunctionsByName(functionName: String): List<AstNode> { if (description == "functionDeclaration") { if (this is AstNode) { children.forEach { if (it.description == "simpleIdentifier" && it.tree() is KlassIdentifier && ((it.tree() as KlassIdentifier).identifier == functionName)) { return listOf(this) + children.flatMap { child -> child.extractAllFunctionsByName(functionName) } } } } else { throw ParsingInterpretationException("wat") } } return if (this is AstNode) { children.flatMap { child -> child.extractAllFunctionsByName(functionName) } } else { listOf() } } private fun Ast.tree(): Ast? { var returned: Ast? = null this.summary(false).onSuccess { returned = it.firstOrNull() } return returned } private fun AstSource.parse() = KotlinGrammarAntlrKotlinParser.parseKotlinFile(this) private fun File.parse(): Ast { val inputStream: InputStream = this.inputStream() val text = inputStream.bufferedReader().use { it.readText() } return KotlinGrammarAntlrKotlinParser.parseKotlinFile(AstSource.String(this.path, text)) } } 
