Heuristics for Improving

Transkript

1 Heuristics for Improving Model Learning Based Testing Muhammad Naeem Irfan VASCO-LIG LIG, Computer Science Lab, Grenoble Universities, Saint Martin d Hères France

2 Introduction Component Based Software Engineering Formal models of such components are missing Methods to construct the model from source code are available Source code for components (black boxes) is most often missing Construct the model by sending inputs and observing outputs

3 Introduction Reverse engineering i models Black Box Observe outputs for inputs Test model Learner Tester Learn model i/o i/o State 1 State 2 State 3 i/o Learned Model

4 Angluin Style Learning Observe outputs Input set I Algorithm Send inputs Perform test OK or Counterexample Learnt Model Oracle Complexity O( I ² nm + I m n²) I : the size of the input set n : the number of states in the actual machine m : the length of the longest counterexample Method by Shahbaz to process the counter example O( I ² n + I mn²)

5 Problem Statement Observe outputs Input set I Algorithm Send inputs Perform test OK or Counterexample Learnt Model Oracle Oracle is of low quality The length of counterexample m is important parameter for the complexity How we can reduce relying on the quality of counterexample Process the counterexample more efficiently

6 Learning Black Box Discriminating Sequences b/y a/x a b State S M s Equ Stat ivalent es S M.I ε a b x x x y y y Black Box q 0 q 2 b/y b/x a/x q 1 a/x Properties: Closed Consistent Observation Table Output Queries s.d, s (St EqSt), d (DescSeq) = a/x ε : an empty string

7 Constructing the Conjecture S M = states, S M.I = eq. states E M = disc.seq seq&t M = output fun Rows Equivalence r 1, r 2 (S M S M.I) are equivalent, iff e E M, T M (r 1,e)=T M (r 2,e) Closeness iff r 1 SS M.I r 2 SS M (r 1 EM r 2 ) Consistency iff i I if (r 1 EM r 2 ) (r 1. i EM r 2. i) ) Black Box b/y a/x q 0 q 2 b/y b/x a/x a/x q 1 Counterexample: Conjecture: Black Box: ababaab a b a a b x y x y x x y x y x y x x x Conjecture q 0

8 Processing the Counterexample Counterexample: a babaab b baab a a b Discriminating Sequences Discriminating Sequences a b ε x y a x y b x y States Eq. States a b ε x y a x y b x y Observation Table (Shahbaz Method) Observation Table (New Method)

9 Difference between Algorithms Processing the Counterexample a b a b a a b Add suffixes to Discriminating Sequences until new state is identified 49 a b ab aab baab abaab babaab 21 a b ab ε x y xy xxx yxxx xyxxx yxyxxx a x y xx xxx yxxx xxxxx yxyxxx aa x x xx xxx xxxx xxxxx xxxxxx b x y xy xxx yxxx xyxxx yxyxxx ab x y xy xxx yxxx xyxxx yxyxxx aaa x x xx xxx xxxx xxxxx xxxxxx Observation Table after processing counterexample with Shahbaz method ε x y xy a x y xx aa x x xx b x y xy ab x y xy aaa x x xx aab x y xx xxx yxxx xxxxx yxyxxx aab x y xx Observation Table after processing counterexample

10 Edinburg Concurrency Workbench Results CWB Examples Tests Examples New Method Shahbaz Method CE New Method CE Shahbaz Method Buff Peterson VM New Method Prev Method VMnew Sched CSPROT System Gain for number of tests conducted Lose for the number of counter examples processed Counterexamples New Method Prev Method

11 Randomly Generated Machines Results Results for learning randomly generated machines with alphabet size 3 and output size 2 for All the Variants of Learning Algorithms State t Size Angluin Method Shahbaz Method New Method , , , ,4 294,5 97, ,4 215,6 120, , , ,7 269, , , , , , ,5 459, , , , , ,5 738, , , , , , , , , , , , ,5 Nu umber of Queries No. of Queries in Angluin Method No. of Queries in Shahbaz Method No. of Queries in first sequence making table unclosed Number of States

12 Comparison of Methods New method less number of discriminating sequences Shahbaz and Angluin method consumes all the discriminatingi i sequences Worst case complexity for Shahbaz and New method is same Gain for processing the Counterexample from Random Gain for processing the Counterexample from Random Walk method

13 Perspectives Find efficient heuristics to bring down the average complexity of algorithm Process the counterexamples How to deal with the abstractions to be made on actual interactions with black box components How to perform random walks more efficiently to obtain the counterexamples Enable learning algorithms to learn implementations as parameterized machines and NFSMs

14 Questions