scenPROC v2 crash

[Clutch Cargo]

Hey Arno,

After editing my spc file to the new 2.0 specs my process crashed at first attempt. The entire script after re-worked is not showing a single "ERROR" message. It crashed at the first attempt to:

CreateAGNLibObj Information Creating autogen library objects with filter FROMFILE="SOCAL_CU63_barn.shp" AND id="0"...



Crash error message:

12:21 PM SceneryProcessor Error One or more errors occurred.
12:21 PM SceneryProcessor Error at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, Action`1 body)
at ASToFra.scenProc.CreateAGNLibObj.Process(List`1 cells, String[] arguments)
at ASToFra.scenProc.SceneryProcessor.Process()
at ASToFra.scenProc.SceneryProcessor.ProcessFile(String filename, Boolean errorReporting)
12:21 PM SceneryProcessor Error CompareElement: Operation 'Equal' is not defined for types 'Int32' and 'String'
12:21 PM SceneryProcessor Error at Ciloci.Flee.ExpressionElement.ThrowCompileException(String messageKey, CompileExceptionReason reason, Object[] arguments)
at Ciloci.Flee.BinaryExpressionElement.CreateElement(IList childValues, Type elementType)
at Ciloci.Flee.FleeExpressionAnalyzer.ExitCompareExpression(Production node)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.Parser.ExitNode(Node node)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.RecursiveDescentParser.ParseAlternative(ProductionPatternAlternative alt)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.RecursiveDescentParser.ParseElement(Production node, ProductionPatternElement elem)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.RecursiveDescentParser.ParseAlternative(ProductionPatternAlternative alt)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.RecursiveDescentParser.ParseElement(Production node, ProductionPatternElement elem)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.RecursiveDescentParser.ParseAlternative(ProductionPatternAlternative alt)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.RecursiveDescentParser.ParseElement(Production node, ProductionPatternElement elem)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.RecursiveDescentParser.ParseAlternative(ProductionPatternAlternative alt)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.RecursiveDescentParser.ParseElement(Production node, ProductionPatternElement elem)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.RecursiveDescentParser.ParseAlternative(ProductionPatternAlternative alt)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.RecursiveDescentParser.ParseElement(Production node, ProductionPatternElement elem)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.RecursiveDescentParser.ParseAlternative(ProductionPatternAlternative alt)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.RecursiveDescentParser.ParseElement(Production node, ProductionPatternElement elem)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.RecursiveDescentParser.ParseAlternative(ProductionPatternAlternative alt)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.RecursiveDescentParser.ParseElement(Production node, ProductionPatternElement elem)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.RecursiveDescentParser.ParseAlternative(ProductionPatternAlternative alt)
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.RecursiveDescentParser.ParseStart()
at Ciloci.Flee.PerCederberg.Grammatica.Runtime.Parser.Parse()
at Ciloci.Flee.ExpressionContext.DoParse()
at Ciloci.Flee.ExpressionContext.Parse(String expression, IServiceProvider services)
at Ciloci.Flee.Expression`1.Compile(String expression, ExpressionOptions options)
at Ciloci.Flee.Expression`1..ctor(String expression, ExpressionContext context, Boolean isGeneric)
at ASToFra.scenProc.FilterEvaluator..ctor(String filter, FilterObjectType type)
at ASToFra.scenProc.GridCell.FilteredFeatures(String filter)
at ASToFra.scenProc.CreateAGNLibObj.<>c__DisplayClass1.<Process>b__0(GridCell cell)
at System.Threading.Tasks.Parallel.<>c__DisplayClassf`1.<ForWorker>b__c()
at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
at System.Threading.Tasks.Task.<>c__DisplayClass11.<ExecuteSelfReplicating>b__10(Object param0)

 

[MyPC8MyBrain]

verify there are no redundant spaces anywhere on your scrip lines!

 

[Clutch Cargo]

You lost me... do you mean " "? Literally blank spaces?

 

[MyPC8MyBrain]

yep
im still testingv2 myself,

this will with similar error
ExportSHP|*|Test| F:\Geo
vs this which completes
ExportSHP|*|Test|F:\Geo

 

[arno]

If I look at the error message the problem is that a string and integer value are compared.

So this is what we discussed earlier in the other thread. The validation only catches those things if you first run scan attributes.

I'll check how I can make it more robust.

 

[Clutch Cargo]

Do you mean scenPROC is getting confused with id="0" as to whether is is a string or integer? If so, issue was not/is not happening with v1.1.

 

[arno]

Hi,

scenProc 1.1 only knew string attributes, so it could not be confused by the type. Now in scenProc 2.0 you need to make sure that the filter you type uses the correct types. I have added an error log message now when something gets wrong, that's better than a crash.

But assuming you have an attribute name type that is a string and width that is an integer. You can't type width="something", because an integer and string can't be compared. So you would need to type width=3. The same goes the other way around type=3 would not work, but type="3" would work.

So I don't know your data, but if id is an integer, then you would have to remove the quotes.

If you press the scan attributes function scenProc will scan your data and be aware of the attribute types defined in the data. In that case the validation will already tell you the filter is not valid. But without scanning the data scenProc doesn't know, so the error only happens during running the script.

 

[Clutch Cargo]

Well, that would make sense as it is an integer. That's probably why none of my scripts were working the first time around. Funny, thought I had it the other way around when testing and it was still throwing an error but need to re-write it again as it was not saved. No biggie. I know this new version could be quite powerful. Just need to get my arms wrapped around it.

 

[MyPC8MyBrain]

just a suggestion -> "string" vs 'integer'

 

[iangbusa]

...So this is what we discussed earlier in the other thread...

This is one of the issues that concerned me when I mentioned use of data types in that thread. I wondered, moving away from only using Strings, how you had implemented them (and how comparison would work). The others were (as mentioned) comparison, results from operations, and also memory use.

Is there, in fact, the possibility of extending what's there to include types like Boolean, float not double, short/long integer, or others? I imagine this depends upon how you've implemented them, used them in operations, or their use memory.

Is the type implementation entirely native to scenproc itself and defined by how you're parsing the steps? Particularly, is each one defined and mapping directly to a C# native data or value type: decimal, or int (only guessing as 64-bit double and long perhaps unnecessary and would take extra memory...) plus Strings "*" (as reference type and char '*' as value type if thought useful)? Perhaps numeric values should assume either int or decimal, depending upon the presence of a decimal point; strings (I'm not convinced chars are necessary) would be assumed if double quotes are used. Other values should be flagged as errors. Can whitespace be ignored when you parse the line unless included in quotes?

Actually, though, it works just fine as-is and users perhaps just need to become more diligent in their coding.

ian

 

[MyPC8MyBrain]

Actually, though, it works just fine as-is and users perhaps just need to become more diligent in their coding.

ian

i tend to agree with your closing statement,
i dabbled myself; but still found my way out on my own relatively easy,
little confusion with the autocomplete there and few other changes require adjustment,
at first i wasn't sure what to do; after some very basic observations with following Arno's instructions in the manual; it made sense,

after that curve i actually started seeing more sense in SQL native statement style; very useful and good for practice,
once i paid closer attention to the changes i needed to make and the new variables;
it is the same as working with previous version only better this time!
(i think its grown a stash now )

 

[arno]

Hi Ian,

For the moment only int, double and string are supported as attribute types. But the underlaying library I use for the expression evaluation supports any type that C# can support. For the moment I don't see much need to add other types though. On one hand OGR (used for reading the data) also has a fixed set of attribute types, for example a bool is not part of that set. If there is a good need in the future we can always add more.

To dive even a little deeper into the code, the strings are supported using a special class I made, because with the normal string I could not get the wildcard search filters to work in the expression evaluator. So I made my own string class to handle those correctly. But as a user you should not notice these details hopefully, unless I added some bugs

I now use double instead of float. I don't think using float will save much memory. Before everything was stored as a string, so a long floating point number would be a string with quite some characters. Still now these are stored as a double I don't see a huge decrease in memory usage. So going to float instead of double will not matter much I expect.

Hope this little inside helps a bit in understanding what is going on,

 

[iangbusa]

Yes, thanks, Arno! Really appreciate all this information to help understand. Indeed, the only type I can see ever becoming useful at some point might be bool as part of the conditional processing of filters but it's certainly not essential. It's easy to use the existing types to fulfill the same functionality. And you never know, I may very well come across the need for a double rather than float value at some point!!

ian

(PS, as you know, my only thought on the double question primarily was the potential of having to use exactly twice as much memory than floats for very large sets of ATTRIBUTES but I hadn't thought about the fact that you'd been using string in the first place. Also, I work with enormous amounts of data in my src files and have never hit a memory problem!)